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ABSTRACT 


This  research  provides  the  first  theoretical  model  —  the  Intranet  Efficiency  and 
Effectiveness  Model  (lEEM)  —  for  the  Eamily  of  Measures  approach  to  measure  Web 
activity  as  well  as  a  holistic  framework  and  multi-disciplinary  quality  paradigm  approach 
not  previously  derived  in  viewing  and  measuring  intranet  contributions  in  the  context  of  a 
corporation’s  overall  critical  business  requirements.  This  is  accomplished  by  applying  a 
balanced  baseline  set  of  metrics  and  conversion  ratios  linked  to  business  processes  as 
they  relate  to  knowledge  workers,  IT  managers  and  business  decision  makers  seeking  to 
increase  value.  It  also  outlines  who  should  conduct  these  measurements  and  how  in  the 
form  of  a  business  intelligence  team  and  provides  a  means  in  which  to  calculate  return  on 
intranet  metrics  investment  (ROIMI)  with  a  common  unit  of  analysis  for  both  aggregate 
and  sub-corporate  levels  through  forms  of  the  Knowledge  Value  Added  (KVA)  and 
Activity  Based  Costing  (ABC)  methodologies. 
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EXECUTIVE  SUMMARY 


Under  current  and  projected  growth  rates  of  information  stored  in  corporate 
intranets  and  the  increasing  need  to  determine  how  valuable  new  portals  are  in  collecting 
and  applying  information  contained  to  meet  specific  business  needs,  employing  a  method  to 
holistically,  uniformly  and  regularly  measure  improvement  and  take  related  actions  to 
effectively  optimize  these  portals  is  of  mounting  importance.  As  IT  professionals  and 
business  decision  makers  seek  ways  to  forge  their  information  into  knowledge  capital  that 
can  be  leveraged  quickly  for  competitive  advantage,  they  require  a  model  and  supporting 
metrics  to  do  so.  Across  any  give  corporation  today,  most  intranet  portal  measurements  are 
based  almost  exclusively  on  usage  statistics  -  with  little  or  no  thought  given  to  design  and 
user  experience  factors  -  and  are  applied  in  a  freelance  and  non-standardized  manner, 
providing  no  meaningful  insight  into  how  well  intranets  help  corporations  achieve  their 
strategic  objectives.  What  has  been  missing  is  a  comprehensive  model  and  methodology  to 
base  measurements  from  logically  related  groups  of  metrics  which,  when  measured 
periodically,  provide  actionable  steps  to  optimize  efficiency  and  effectiveness  of  intranet 
portals  to  better  bolster  key  business  requirements  in  pursuit  of  value.  This  research 
provides  the  first  theoretical  model  for  the  Family  of  Measures  approach  to  measure  Web 
activity  as  well  as  a  holistic  framework  and  multi-disciplinary  quality  paradigm  approach 
not  previously  derived  in  viewing  and  measuring  intranet  contributions  in  the  context  of  a 
corporation’s  overall  critical  business  requirements.  This  is  accomplished  by  applying  a 
balanced  baseline  set  of  metrics  and  conversion  ratios  linked  to  business  processes  as  they 
relate  to  knowledge  workers,  IT  managers  and  business  decision  makers  seeking  to  increase 
value.  It  also  outlines  who  should  conduct  these  measurements  and  how  in  the  form  of  a 
business  intelligence  team  and  provides  a  means  in  which  to  calculate  return  on  intranet 
metrics  investment  (ROIMI)  with  a  common  unit  of  analysis  for  both  aggregate  and  sub¬ 
corporate  levels  through  forms  of  the  Knowledge  Value  Added  (KVA)  and  Activity  Based 
Costing  (ABC)  methodologies.  Corporations  that  use  the  holistic  six  segment  perspectives 
outlined  in  the  Intranet  Efficiency  and  Effectiveness  Model  (lEEM)  to  define,  apply  and 
refine  a  balanced  set  of  metrics  and  conversion  rates  will  measure  and  take  action  on  what  is 
important,  instead  of  what  is  available. 
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I.  INTRODUCTION 


Much  relevant  work  in  other  disciplines  such  as  finance,  economics 
and  decision  theory  are  seldom  encountered  by  the  typical  software  engineer.  An 
interdisciplinary  approach  that  integrates  such  work  with  traditional  software 
engineering  theory  and  practice  promises  real  advantages.  The  purpose  of 
software  design,  as  for  any  design  activity,  is  to  create  added  value.  Y et,  current 
theories,  concepts,  tools  and  methods  are  not  clearly  based  on  modem  models  of 
value  and  value  creation. . . 

Manifesto,  Economics  Driven  Software  Engineering  Research  (EDSER) 

A,  BACKGROUND 

The  corporate  portal  is  one  of  the  earliest  broad  applications  of  information 
technology  (IT)  to  immediately  impact  knowledge  management  practice.  Corporate 
portals  have  a  purpose  that  is  similar  to,  but  fundamentally  distinct  from,  Internet  portals 
that  have  become  the  popular  interface  for  searching  and  traversing  the  World  Wide 
Web.  Corporate  portals  provide  an  architecture  and  set  of  technologies  with  which  to 
build  a  single  point  of  access  across  a  wide  variety  of  applications,  repositories, 
processes,  and  functions  that  have  proliferated  in  the  corporate  information  environment. 
The  portal  market  can  be  divided  into  four  broad  categories:  business  to  employee, 
business  to  consumer,  business  to  supplier,  and  personalized  Web  front-ends,  like  Yahoo. 
This  paper  focuses  on  portals  designed  for  business  to  employee,  known  as  intranets. 

Most  enterprises  today  find  themselves  poorly  positioned  to  take  competitive 
advantage  of  the  recent  proliferation  of  corporate  information  sources  resulting  from  the 
advent  of  the  intranet.  A  fundamental  shortcoming  in  doing  this  is  due  to  a  lack  of 
comprehensive  and  credible  means  in  which  to  measure  how  effective  the  portals  meet 
the  demands  of  their  employees  (also  known  as  knowledge  workers)  and  other  intended 
audiences  in  pursuit  of  carrying  out  business  objectives. 

Intranets  exist  to  fulfill  different  purposes  for  different  constituencies  than  does 
the  Internet.  The  key  difference  lies  in  the  underlying  mission  of  the  portal  itself:  on  the 
Internet,  the  portal  sites’  business  model  is  based  on  attracting  a  portion  of  the  advertising 
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budgets  of  corporations  that  might  otherwise  advertise  in  other  media  (print,  TV,  radio, 
etc.).  Thus,  the  general  purpose  of  the  public  portals  is  to  attract  large  numbers  of  repeat 
visitors,  to  build  online  audiences  with  the  inclination  to  buy  what  the  portal  advertisers 
have  to  sell.  These  portals  have  essentially  settled  into  a  one-directional  relationship  with 
their  viewers. 

Inside  the  organization,  the  portal  takes  on  an  entirely  different  character.  It  takes 
its  purpose  from  the  overall  mission  of  the  organization;  to  add  sufficient  value  for  its 
customers  to  create  a  sustainable  business  model.  It  takes  its  features  and  functionality 
from  the  mandate  to  operate  at  world-class  efficiency  and  effectiveness  in  order  to  remain 
competitive.  Achieving  this  competitiveness  requires  a  bi-directional  model  that  can 
support  the  knowledge  workers’  increasingly  sensitive  needs  for  pertinent,  helpful,  timely 
content  and  interactive  information  management  tools. 

The  primary  functionality  of  a  portal  is  in  its  ability  to  select,  filter,  expose,  and 
deliver  with  a  significant  degree  of  precision  the  information  that  knowledge  workers 
within  an  organization  require  to  pursue  core  business  processes  and  competencies  in 
support  of  strategic  business  requirements.  Unlike  publication  portals  or  commercial 
portals  which  serve  essentially  as  one  way  bulleting  boards,  the  corporate  portal  has  its 
information  context  predetermined  by  the  information  requirements  and  applications 
already  in  place  in  the  organization.  In  this  respect,  corporate  portals  share  some  of  the 
built-in  context  which  gives  distinctive  shape  to  departmental  and  group  portals 
supporting  knowledge  workers  within  or  partners  outside  who  share  similar  job  functions 
and  objectives.  The  opportunity  for  corporate  portal  developers  is  to  identify  and  tap  into 
the  underlying  organization  in  disparate  corporate  information,  which,  if  correctly 
exploited,  can  yield  significant  benefits  for  knowledge  workers  who  are  tasked  with 
navigating  the  myriad  internal  and  external  sources  of  connections  among  people, 
processes,  and  the  information  sources.  As  organizations  realize  their  customer  data  and 
the  intellect  of  their  knowledge  workers  are  their  greatest  assets,  intranets  will  grow  and 
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their  portals  will  be  better  interconneeted  through  inereased  emphasis  on  taxonomy  and 
tagging  management  in  support  of  more  relative  and  faster  seareh  and  navigational  finds. 
The  goal  is  to  provide  virtually  all  the  information  knowledge  workers  need  at  their 
fingertips  to  be  highly  productive  and  to  sustain  competitiveness. 

Portals  will  become  a  competitive  necessity  in  many  sectors  without  which  a 
corporation  would  be  at  a  serious  disadvantage.  Content  which  once  was  lost  in  a  sea  of 
intranet  websites,  obsolete  navigation  hierarchies,  or  legacy  data  stores  will  one  day  be 
integrated  into  a  single  and  more  personalized  experience  —  dramatically  increasing  the 
quality  of  results  made  and  thereby  improving  the  odds  that  more  can  be  accomplished 
with  them.  This,  by  inference,  leads  to  increases  in  productivity  and  greater  value. 
Corporate  intranets  built  today  will  eventually  be  logically  brought  together,  optimized 
and  deployed  correctly  making  them,  in  effect,  the  backbone  of  the  knowledge  portals  of 
tomorrow. 

B,  MOTIVATION 

As  organizations  continue  to  consume  massive  amounts  of  data,  the  number  of 
portals  deployed  gradually  increases  to  personalize  and  meet  the  needs  of  specific  groups. 
In  fact,  the  primary  impetus  behind  portal  adoption  is  deep  user  dissatisfaction  with  a 
one-size-fits-all  Internet  as  well  as  unwieldy  intranets.  Usually  the  larger  an  intranet 
grows  the  inherently  more  inefficient  and  ineffective  it  becomes,  further  inhibiting 
information  from  becoming  knowledge.  Nevertheless,  there  is  great  potential  to 
transform  large  intranets  into  productivity  applications  in  their  own  right  and  a  growing 
number  of  corporations  are  using  portals  to  begin  turning  this  into  a  reality.  As  a  result, 
the  critical  work  of  software  engineering  in  applying  data  abstraction  as  well  as  software 
architecture  design  and  support  required  to  build  efficient  and  effective  intranets  is 
playing  an  incontrovertible  and  significant  role. 
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The  primary  objective  of  creating  Intranet  portals  is  to  create  value  through 
quality  improvements  in  productivity  and  customer  satisfaction  by  the  exchange  of 
desirable  business  related  knowledge.  Recent  interest  in  the  software  engineering 
community  in  portal  processes  to  leverage  knowledge  reflects  this  aspect  of  the  quality 
paradigm.  One  fundamentally  unique  way  of  seeing  and  solving  problems  in  the  quality 
paradigm  is  to  focus  on  processes  rather  than  products,  services,  mistakes,  errors,  or 
traffic  volume.  Work  in  the  quality  paradigm  deals  with  recognizing,  defining, 
measuring,  analyzing  and  improving  processes.  A  central  value  of  the  quality  paradigm 
is  that  processes  must  not  merely  be  improved  but  they  must  be  continually  improved. 
The  same  holds  true  in  measuring  and  improving  value  created  by  intranet  portals.  Only 
those  organizations  that  continue  to  refine  and  measure  the  subsequent  impacts  of  their 
portals  will  realize  greater  value  gains. 

The  value  portals  apparently  already  bring  to  enterprises  is  strongly  evidenced  by 
the  recent  surge  in  portal  investments'.  Corporations  of  all  kinds  expect  big  benefits  such 
as  better  access  to  information  —  but  large  majorities  of  them  say  they  do  not  measure 
intranet  benefits  because  they  are  too  abstract  and  don't  convey  the  impact  that  their 
portals  have  on  overall  corporate  performance.  Nevertheless,  knowing  how  intangible 
assets  affect  performance  can  mean  the  difference  between  growth  and  erosion  of  value. 
Intranet  analytics  is  not  an  oxymoron  as  perceived  by  many,  but  it  can  be  counter¬ 
intuitive  and  more  research  should  be  devoted  to  this  area  to  lead  the  way  in  how  to  seize 
considerable  new  opportunities. 

Presently,  most  IT  organizations  today  do  not  have  clear  ways  to  calculate  and 
communicate  the  direct  benefits  produced.  Similarly,  the  software  engineer  community 
lacks  deep  theoretical  understanding  of  the  means  by  which  core  concepts  of  their  field 
are  linked  to  value  creation,  e.g.,  modularity  in  design  (architecture),  iterative 

*  According  to  one  research  firm,  60  percent  of  Global  2000  companies  will  have  at  least  a  first-generation 
employee  portal  this  year  and  most  of  these  portals  will  be  designed  primarily  for  employees'  use 
(Anderson,  pp.  43-57). 
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development  methods,  testing,  ete.  Thus,  the  eurrent  coneeptual  state  of  the  art  in 
software  design  and  engineering  is  not  elearly  optimal  for  value.  To  the  extent  that  it  is 
not,  society  is  not  getting  its  money’s  worth  from  its  software  engineers.  The  emerging 
area  of  strategic  analysis,  software  and  domain  design,  undertaken  in  this  research,  takes 
the  value -based  view  as  a  premise  and  seeks  to  develop  and  enhance  fundamental 
software  architecture  and  design  theories,  tools  and  methods  by  basing  them  on  emerging 
models  of  value  and  value  creation  which  are  supported  by  both  quantitative  and 
qualitative  metrics  groupings. 


Less  - 


More 


Work  to  date  in 
IT  community 


Strategic  Management 


Economic  Valuation  /' 


'\Costs  &  Benefits  ' 


''  Metrics/ 


Knowledge  Workers 
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Figure  1,  Software  Engineering  Approach  to  Metrics 


Information  technology  contributes  more  than  data;  it  impacts  a  wide  range  of 
business  processes  that  produce  the  required  results  to  sustain  value.  It  is  in  the  analysis 
of  these  processes  that  answers  to  productivity  can  be  derived.  However,  in  most 
corporations  there  is  no  consistent,  visible  and  uniform  way  to  measure,  manage  and 
maximize  the  effectiveness  of  IT-enabled  business  processes  of  intranets.  The  motivation 
for  this  research  is  to  provide  a  model,  {Intranet  Efficiency  and  Effectiveness  Model, 
lEEM),  and  means  of  measuring  the  effectiveness  of  a  processes  integrated  approach  to 
quality  through  the  establishment  and  recommended  application  of  a  baseline  set  of 

metrics  and  their  conversions  ratios  that  collectively  and  logically  relate  to  key  strategic 

5 


business  requirements  that  drive  value.  Hypothetieally,  ineremental  results  are 
aehievable.  This  paper  further  outlines  who  should  do  the  analyties  and  how  in  order  for 
eorporations  to  move  forward  with  a  eontinuous  optimization  proeess. 


C.  CONCEPTUAL  SCOPE  OF  RESEARCH 

Taking  an  interdiseiplinary  approaeh  to  this  matter  reveals  the  underlying 
segments  that  eonstitute  an  intranet  and  their  dependeneies.  This  ean  then  be  delineated, 
resulting  in  logieal  eonstituents  that  ean  be  measured  with  groups  of  related  metries  using 
multiple  tools  and  teehniques.  The  supposition  is  derived  metries  that  indieate 
effeetiveness  of  a  eorporation’s  intranet  portals  are  aehievable  and  ean  be  presented  to 
provide  meaningful  feedbaek  as  well  as  aetionable  steps  to  further  improve  portal 
operations  by  taking  into  aeeount  and  uniformly  applying  aeross  all  portals,  a  more 
holistie  and  eoherent  eombination  of  metries  from  all  intranet  segments.  The  end  result 
and  goal  is  to  drive  more  value  baek  into  an  intranet  than  would  be  garnered  otherwise 
through  more  preeision  tuning  of  portals  in  support  of  business  requirements  based  on 
audienee  needs  and  behavior  as  well  as  usage  traflie  figures  and  eonversion  rates. 

D,  RESEARCH  CONTRIBUTION 

Although  most  intranet  portal  owners  agree  in  prineiple  that  measurement  is 
important  for  sueeess,  many  fail  to  properly  implement  a  portal  management  effort  unless 
budget  justifieations  are  required.  Indeed,  researeh  indieates  that  76%  of  Global  2000 
firms  do  not  eonduet  benehmarking  even  onee  a  year. 

•  only  19%  of  Global  2000  firms  eonduet  it  annually. 

•  just  14%  eontinually  refresh  benehmarks  and  use  the  results  regularly  in 
performanee  reporting  (Roth,  2002). 

There  is  a  woeful  laek  of  forethought  in  seleeting  metries  for  intranets,  solieiting  foeused 
and  “impaetful”  surveys  for  feedbaek  or  how  to  do  appropriate  analysis  with  it  to  provide 
aetionable  steps  to  improve  portal  effeetiveness.  Aeross  any  given  eorporation,  most 
intranet  portal  measurements  today  are  based  almost  exclusively  on  usage  statistics  - 
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with  little  or  no  thought  given  to  design  and  user  experience  faetors  -  and  are  applied  in  a 
freelanee  and  non-standardized  manner.  While  these  metries  provide  a  basis  to 
benehmark  against  eompetitors  or  other  departments/groups  within  the  organization,  they 
provide  no  meaningful  insight  into  how  well  businesses  are  aehieving  their  strategie 
objeetives. 

As  more  and  more  portal  mangers  seek  a  means  to  ensure  portals  are  being 
properly  deployed  and  modified  in  support  of  their  employees,  their  sponsoring  business 
deeision  makers  (BDMs)  will  seareh  for  a  strategy  to  prove  its  sueeess,  and  not  one  based 
strietly  on  conventional  return  on  investment  (ROI).  Portal  costs  are  akin  to  sunk  costs 
and  the  actual  costs  to  improve  them  usually  only  require  analytical  capabilities 
supported  by  minor  and  affordable  modifications  to  the  system  and  not  costly,  major 
hardware  and  software  developments  that  require  long  stretches  of  time. 

The  most  meaningful  way  to  measure  an  intranet  is  by  the  effectiveness  of  its 
collective  portals  in  support  of  business  requirements.  The  delineation  between  costs  and 
revenue  versus  effectiveness  and  productivity  —  along  with  the  lack  of  any 
straightforward  model-based  method  to  indicate  the  increase  in  the  potentiality  of  events 
that  lead  to  productivity  gains  —  is  the  reason  why  measuring  intranets  are  often 
overlooked  and  misunderstood.  The  research  surrounding  the  Intranet  Efficiency  and 
Effectiveness  Model  contributes  to  the  methodology  and  analytic  basis  needed  to  select  a 
baseline  group  of  metrics  and  conversion  ratios  that  help  to  logically  and  holistically 
prove  out  and  optimize  effectiveness  of  intranet  portals  as  they  relate  to  and  bolster 
strategic  business  requirements  which  steer  the  productivity  factors.  An  intention  being 
corporations  can  use  the  six  segment  perspectives  outlined  in  the  lEEM  to  define,  apply 
and  refine  a  balanced  set  of  metrics  to  begin  measuring  what  is  important,  instead  of  what 
is  available. 
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Though  credit  is  due  other  perspectives,  meeting  the  business  requirements  is  the 
bottom  line  to  measuring  the  effectiveness  of  any  website.  Within  the  total  set  of  metrics, 
there  needs  to  be  some  monetary  business-based  metrics,  such  as  ROI,  customer  lifetime 
value,  customer  recency,  conversion  rates,  and  eross-portal  referrals  —  depending 
naturally  on  the  specific  business  objectives  of  the  corporation’s  overall  eustomer 
relationship  and  management  strategy.  To  ealculate  these  metrics,  corporations  need  to 
integrate  data  from  many  sources,  possibly  spanning  the  complete  enterprise.  This  can  be 
quite  ehallenging,  but  in  the  end  organizations  need  these  metrics  as  tangible  feedback  for 
their  efforts  in  creating,  maintaining  and  managing  their  Web  portals,  partieularly  if  they 
are  required  to  present  IT-related  initiatives  in  finaneial  terms  for  return  on  investment 
analysis. 

To  this  end,  this  research  provides  the  first  theoretieal  model  for  the  Family  of 
Measures  approaeh  to  measure  Web  activity  (Section  II. B:  Review  of  Current  IT  Analysis 
Approaches)  as  well  as  a  holistic  framework  and  multi-diseiplinary  approaeh  not 
previously  derived  in  viewing  and  measuring  intranet  contributions  in  the  context  of  a 
eorporation’s  overall  eritical  business  requirements  by  applying  a  balanced  baseline  set  of 
metrics  and  conversion  rations  —  which  merge  website  traffie  data  as  well  as  user 
behavior  -  that  are  linked  to  business  proeesses  as  they  relate  to  knowledge  workers,  IT 
portal  managers  and  BDMs.  It  also  outlines  who  should  do  this  and  how  in  the  form  of  a 
business  intelligence  team  and  further  provides  a  means  in  whieh  to  calculate  ROI  for  the 
metries  investment  with  a  common  unit  of  analysis  for  both  aggregate  and  sub-corporate 
levels  through  a  form  of  the  Knowledge  Value  Added  (KVA)  and  Activity  Based  Costing 
(ABC)  methodologies. 

E,  WAY  AHEAD 

The  following  ehapters  address  in  sequenee  related  work  and  their  gaps,  the 
coneeptual  framework  of  the  Intranet  Efficiency  and  Effectiveness  Model  introduced, 
how  the  lEEM  is  related  and  applied  to  strategic  business  requirements,  followed  by  how 


a  return  on  investment  can  be  calculated  and  by  whom,  before  concluding  with  a 
summary  and  a  recommendation  for  future  related  work. 


9 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


10 


II.  ASSESSMENT  OF  RELATED  WORK 


The  software  development  proeess  must  ineorporate  more 
and  more  strategie  aspects.  Project  success  is  more  about  whether  the 
software  delivers  value  that's  greater  than  the  cost  of  the  resources  put  into 
it  -  but  that's  very  tricky  to  measure. 

Martin  Fowler,  Enterprise  Software  Consultant 

A,  CURRENT  APPROACHES  TO  WEB  MEASUREMENTS 

The  primary  reason  for  most  investments  in  information  technology  is  to  improve 
business  processes.  The  problem  becomes  one  of  discerning  how  much  value  the  IT  will 
add  to  the  processes.  One  way  to  answer  to  this  question  would  be  to  determine  how 
much  return  the  IT  provides  at  the  aggregate  (referred  to  hereafter  as  corporate)  and  sub¬ 
corporate  levels.  There  have  been  numerous  approaches  to  assessing  the  impact  of  IT  on 
company  economic  performance  at  the  corporate  level  of  aggregation  and  sub-corporate 
levels  (Brynjolfsson  and  Yang,  1996).  The  lEEM  is  the  first  theoretical  framework  that 
can  comprehensively  unify  analysis  to  address  this  problem,  regardless  of  level  of 
aggregation. 

Although  a  great  deal  has  been  written  on  how  to  calculate  return  of  investments 
for  the  Internet,  there  is  an  appreciable  lack  of  data  on  how  to  measure  the  effectiveness 
of  an  intranet.  Current  IT  measurements  either  focus  predominantly  on  online  volume 
traffic  or  its  ROI  based  on  some  monetary  cost  allocation  calculation.  The  following 
table,  taken  from  Models  for  Measuring  the  Return  on  Information  Technology  by  Dr. 
Thomas  Housel  (Housel  et  ah,  2001,  p.  4),  provides  a  snapshot  of  some  of  the 
predominant,  current  approaches.  This  proposal,  A  Metric  Model  for  Intranet  Portal 
Business  Requirements,  is  appended  at  the  end  to  provide  a  basis  of  comparison  and  to 
highlight  its  strengths  over  the  other  methodologies  presented.  A  review  of  these 
approaches  follows  which  identifies  a  number  of  problematic  issues  and  limitations  of 
each. 
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Approach 

Focus 

Example 

Level  of 
Analysis 

Key 

Assumption 

Key 

Advantage 

Limitation 

Process  Of 
Elimination 

Treats  effect  of  IT 
on  ROI  as  a 
residual  after 
accounting  for 
other  more  easily 
measurable  capital 
investment 

Knowledge 
Capital 
(Strassmann 
2000a,  b) 

Aggregate 
corporate  -level 
only 

ROI  on  IT 
difficult  to 

measure 

directly 

Uses  commonly 

accepted 

financial 

analysis 

techniques  and 

existing 

accounting  data 

Cannot  drill 
down  to  effects 
of  specific  IT 
initiatives 

Production 

Theory 

Determines  the 
effects  of  IT 
through  input 
output  analysis 
using  regression 
modeling 
techniques 

Brynjolfsson 
&  Hitt  (1996) 

Aggregate 
Corporate  -  level 
only 

Economic 
Production 
Function  Links 

IT  Investment 
Input  To 
Productivity 
Output 

Uses  Econo¬ 
metric  Analysis 
on  Large  Data 
Sets  to  Shows 
Contributions 
of  IT  at  the 

Firm  Level 

"Black-Box" 
approach  with  no 
intermediate 
mapping  of  IT's 
contributions  to 
outputs 

Resource- 
Based  View 

Linking  Firm  Core 
capabilities  with 
competitiveness 

Jarvenpaa  & 
Leidner  (1998) 

Aggregate 
Corporate  -  level 
only 

Uniqueness  of 

IT  Resource  = 

Competitive 

Advantage 

Strategic 
advantage 
approach  to  IT 
impacts 

Causal  mapping 
between  IT 
investment  and 
Firm  Competi¬ 
tive  Advantage 
difficult  to 
establish 

Option 

Pricing 

Model 

Determines  the 
best  point  at 
which  to  exercise 
an  option  to  invest 
in  IT 

Benaroch  & 

Kauffman 

(1999) 

Corporate/Sub¬ 

corporate 

Timing 

Exercise  Option 
=  Value 

Predicting  The 
Future  Value  of 
An  IT 

Investment 

No  Surrogate  For 
Revenue  At  Sub 
Corporate  Level 

Family  of 
Measures 

Measure  multiple 
indicators  to 
derive  the  unique 
contributions  of 
information 
technology  at  the 
subcorporate  level 

Balanced 
Score-Card 
(Kaplan  and 
Norton,  1996) 

Sub-corporate 

Need  Multiple 
Indicators  to 
Measure 
Performance 

Captures 
Complexity  of 
Corporate 
Performance 

No  Common 

Unit  of  Analysis, 

No  Theoretical 
Framework 

Cost-Based 

Use  cost  to 
determine  the 
value  of 
information 
technology 

Activity-based 

Costing 

Johnson  & 

Kaplan 

(1987) 

Sub-corporate 

Derivations  of 
Cost  =  Value 

Captures 

Accurate  Cost 
of  IT 

No  Surrogate  For 
Revenue  At  Sub 
Corporate  Level, 
No  Ratio 

Analysis 

Intranet 

Efficiency 

and 

Effective¬ 
ness  Model 

Use  hard  and  soft 
combinations  of 
metrics  from  each 
segment  of 
intranet 

Business 
Requirement 
to  Audience 
Metrics 
(Jacoby  2002) 

Corporate/Sub¬ 

corporate 

Multiple  groups 
of  logical 
metrics 
associated  to 
key  business 
requirements  of 
productivity 

Provides  Theore 
tical  ramework 
and  Surrogate 
for  Revenue  At 
Corporate  and 
Sub  Corporate 
Levels 

Not  all 
subjectivity  in 
analysis  is 
completely 
eliminated 

Table  1,  Common  Approaches  to  Measure  the  Return  on  IT 
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B,  REVIEW  OF  CURRENT  IT  ANALYSIS  APPROACHES 

Research  on  the  problem  of  estimating  the  value  added  by  IT  can  be  categorized 
at  two  levels  of  analysis:  corporate  and  sub-corporate.  At  the  corporate  level  existing 
approaches  can  be  categorized  as:  Process  of  Elimination,  Production  Theory,  and 
Resource-based  View.  Two  of  the  prominent  sub-corporate  approaches  can  be 
categorized  as:  Family  of  Measures  and  Cost-Based.  The  Options  Pricing  Model  and 
lEEM  span  both  levels  in  that  they  can  be  used  to  assess  corporate  level  IT  investments 
or  individual  IT  initiatives. 

1.  Corporate  Analysis 

The  corporate  level  approaches  are  largely  designed  to  help  investors  understand 
the  contribution  of  corporate  assets  such  as  knowledge  and  technology  (e.g.,  see 
Brynjolfsson  and  Hitt  1996,  Brynjolfsson  and  Yang  1996,  Im,  Dow,  and  Grover  2001, 
Strassman  1997)  to  a  firm’s  or  industry  sector’s  performance. 

a.  Process  of  Elimination 

In  the  "process  of  elimination"  approaches,  the  various  costs  for  capital 
(e.g.,  equipment,  real  estate,  raw  materials)  are  removed  leaving  the  cost  of  technology. 
Once  the  costs  for  capital  are  accounted  for  and  income  proportionately  reduced,  the 
residual  is  asserted  to  be  revenue  attributable  to  knowledge  capital  and/or  information 
technology.  Following  this  approach,  all  costs  attributable  to  all  cost  categories,  except 
IT,  would  reduce  the  income  proportionately  leaving  the  income  attributable  to  the  IT. 

The  limitation  of  these  approaches  to  measuring  the  value  added  by  IT  to 
core  processes  is  that  they  only  pertain  to  the  aggregate  view.  It  is  unlikely  that  such 
aggregated  views  would  allow  precise  inferences  about  performance  improvements 
derived  from  IT  initiatives  at  the  process  level  (Birchard  and  Nyberg  2001). 
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b.  Production  Theory 

Others  have  used  eeonomie-based  produetion  theory  to  determine  the 
various  eontributions  of  inputs  to  the  firm’s  output.  Hitt  and  Brynjolfsson  (1995) 
measured  the  value  of  IT  in  terms  of  produetivity,  profit,  and  eonsumer  welfare. 
Extending  this  researeh,  Brynjolfsson  and  Yang  (1996)  report  eomprehensively  on  IT’s 
effeet  on  overall  eompany  performanee.  The  resulting  “produetion  funetion” 
(Brynjolfsson  and  Hitt  1996,  p.  545)  ean  be  modeled  using  eeonomie  theory  to  determine 
the  unique  eontributions  of  IT  with  eomputer  eapital,  non-eomputer  eapital,  information 
systems  staff  labor  and  other  labor  expenses  as  the  inputs  (whieh  represent  all  spending 
by  the  organization  as  well  as  all  eapitalized  investment)  and  output  in  terms  of  dollars  or 
physieal  units.  This  neoelassieal  eeonomie  theory  of  produetion  treats  firms  as  “blaek 
boxes”  (Brynjolfsson  and  Hitt  1996,  p.  544)  and  attempts  to  deduee  the  relationship 
between  inputs  and  outputs  without  referenee  to  aetivities  within  the  eompany. 

The  strength  of  sueh  approaehes  derives  from  their  relianee  on  eommonly 
aeeepted  finaneial-eoonomie  theories  and  the  use  of  existing  aeeounting  data.  This 
makes  them  transparent  for  review  and  eomparison.  However,  various  eritieisms  have 
been  leveled  at  these  approaehes  ineluding  that  the  researeh  using  these  approaehes  does 
not  “adequately  eontrol  for  other  faetors  [i.e.,  other  than  information  teehnology]  that 
drive  firm  profits”  (Bharadwaj  2000,  p.  170).  Along  the  same  lines,  Im  et  al.  (2001,  p. 
104)  stated,  "Beeause  many  faetors  influenee  firm  performanee,  it  is  difficult  to  establish 
causality  between  IT  investments  and  firm-level  output  performance."  This  lack  of 
intermediate  mapping  of  IT  impacts  on  processes  makes  this  class  of  approaches 
problematic  for  providing  the  kinds  of  feedback  necessary  to  help  management  determine 
whether  their  IT  initiatives  are  really  paying  off. 

c.  Resource-Based  View 

Researchers  using  the  resource-based  view  attempt  to  overcome  the 
shortcomings  of  the  prior  methodologies  for  estimating  the  value  added  by  IT  by 
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reformulating  the  problem.  They  attempt  to  link  a  firm’s  performanee  to  IT  resourees 
that  are  firm-speeific  sueh  as  knowledge,  eapabilities  and  unique  core  processes 
(Bharadwaj  2000,  Jarvenpaa  and  Leidner  1998)  .  The  assumption  is  that  these  unique 
resources  and  capabilities  are  difficult,  and  very  expensive,  to  copy  and  therefore  provide 
competitive  advantages  leading  to  superior  economic  returns. 

A  limitation  of  this  view  for  tracking  the  specific  value-added 
contributions  of  IT  is  that  it  does  not  posit  a  common,  granular  unit  of  analysis  that  would 
allow  an  unambiguous  linkage  or  mapping  between  a  firm’s  use  of  IT  and  the  resulting 
cost-benefits  performance.  Using  this  approach,  it  would  be  difficult  to  unambiguously 
determine  the  specific  contribution  of  a  given  IT  initiative. 

d.  Option  Pricing  Models 

The  application  of  option  pricing  models  (0PM)  to  IT  investment  has 
attracted  increasing  attention  (cf  Benaroch  and  Kauffman,  1999).  Basically,  the  0PM 
approach  in  evaluating  IT  initiatives  determines  the  best  point  at  which  to  exercise  an 
option  to  invest.  Applied  in  the  context  of  real  options  investments,  there  are  six 
variables  used  to  make  the  decision:  The  current  value  of  the  underlying  asset. 

•  The  time  to  the  decision  date. 

•  The  investment  cost  or  exercise  price  (also  called  the  strike  price). 

•  The  risk-free  rate  of  interest. 

•  The  volatility  of  the  underlying  asset,  which  is  often  the  only  estimated 

output. 

•  Cash  payouts  or  non-capital  gains  returns  to  holding  the  underlying  assets. 
(Amram  and  Kulatilaka  1999,  37). 

The  0PM  approach  has  some  implicit  assumptions  that  are  potentially 
limiting.  For  example,  net  present  value  is  used  in  the  calculation  of  risk  and  requires  an 
assumption  about  projected  cash  flow.  However,  there  is  no  cash  flow  directly 
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attributable  to  most  corporate  processes.  This  is  because  the  outputs  of  those  processes 
are  not  salable  to  end  customers  without  the  outputs  of  all  the  other  processes,  limiting 
the  applicability  of  this  approach  at  the  sub-corporate  level. 

2,  Sub-Corporate  Analysis 

The  current  aggregate  level  approaches  do  not  appear  to  resolve  the  problem  of 
determining  the  IT  impacts  on  process  performance  (sub-corporate  level)  with  enough 
precision  to  benefit  managers  who  must  implement  changes  at  the  process  level.  Hence, 
other  approaches  for  addressing  problems  of  measuring  the  impact  of  IT  have  focused  at 
the  sub-corporate  level. 

a.  Family  of  Measures 

The  family-of-measures  approaches.  Balanced  Scorecard  and  Intellectual 
Capital  Navigator  advocate  the  need  to  measure  multiple  indicators  to  derive  the  unique 
contributions  of  information  technology  at  the  sub-corporate  level.  The  “Balanced 
Scorecard”  provides  typically  from  four  to  five  key  performance  indicators  selected  by 
management  to  determine  the  success  of  a  given  strategic  organizational  thrust  (Kaplan 
and  Norton  1996).  In  the  case  of  an  IT  initiative,  the  organization’s  management  team 
might  select  this  initiative  for  assessment  via  a  set  of  key  performance  indicators,  for 
example  including  level  of  customer  satisfaction,  financial  performance,  employee 
satisfaction  and  core  process  performance.  Edvinsson  and  Malone’s  (1997)  Intellectual 
Capital  Navigator  allows  a  firm  to  identify  up  to  140  variables  that  account  for  the 
performance  of  its  intangible  assets  including  IT  (i.e.,  a  subset  of  its  infrastructure  assets). 
Examples  of  these  measures  would  include:  laptops/employee,  IT  expense/employee,  IT 
staff/staff  total,  IT  literacy  of  employees,  and  so  on  (Edvinsson  and  Malone’s  1997,  p. 
109). 
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b.  Cost-Based 

Many  of  the  cost-based  approaches  use  underlying  replacement  cost  (e.g., 
transfer  pricing,  internal  markets,  outsourcing)  to  determine  the  value  of  information 
technology  (Housel  and  Bell  2001).  These  approaches  assume  that  the  cost  of  IT  is  in 
some  way  proportionate  to  its  value.  For  example,  the  cost  to  replace  or  outsource  IT  is 
presumed  to  be  proportionate  to  the  value  it  adds  to  process  performance.  Other 
approaches  assume  that  by  introducing  a  market  mechanism  where  corporate  managers 
submit  bids  for  IT  services,  the  resulting  market  price  is  representative  of  the  IT’s 
underlying  value  (Ba,  Stallaert  and  Whinston  2001). 

C.  HOW  THE  lEEM  APPROACH  WOULD  DIFFER  AND  COMPLIMENT 

The  Intranet  Efficiency  and  Effectiveness  Model  views  intranet  portal-related 
initiatives  and  application  of  its  associated  metrics  baseline  and  conversion  ratios  in  the 
context  of  the  organization’s  overall  strategy  by  linking  metrics  groups  to  critical 
business  requirements  and  how  they  effect  and  are  affected  by  knowledge  workers,  portal 
managers  and  business  decision  makers.  Along  these  lines,  the  framework  of  lEEM 
actually  complements  and  provides  the  first  theoretical  model  based  on  all  segments  of 
the  intranet  processes  for  the  Eamily  of  Measures  concept.  In  addition,  the  metrics 
baseline  is  similar  to  that  of  the  Balanced  Scorecard  but  extends  it  by  providing  context 
linked  to  strategic  business  requirements.  The  ROI  for  lEEM  (see  section  V.E:  Objective 
of  ROIMI)  is  similar  to  Cost-Based  in  that  it  is  founded  on  cost-benefit  analysis  with  time 
being  the  common  unit  of  analysis  which  can  be  used  at  both  the  corporate  and  sub¬ 
corporate  levels,  a  faculty  lacking  in  the  other  analysis  approaches. 

The  main  assumption  when  a  corporation  assumes  the  lEEM’s  supporting  metrics 
to  serve  as  a  baseline  to  perform  analytics  is  that  it  already  has  an  intranet  (i.e.,  sunk  cost 
as  a  part  of  doing  business)  and  believes  there  is  room  for  continuous  optimization  in  it  to 
increase  its  net  value.  The  ROI  conjectured  is  essentially  a  cost-benefit  between  the  time 
needed  to  invest  in  applying  and  acting  upon  the  results  of  the  metric  analytics  and  the 
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results  in  time  reduetions  due  to  subsequent  ehanges  introdueed  by  this  analysis  proeess, 
i.e.,  shorter  eompletion  time  of  a  series  of  business  related  tasks  (sub-eorporate  level)  and 
speed  to  market  or  eompletion  of  a  projeet  (eorporate  level). 

1,  Family  of  Measures  Similarities  and  Differences 

The  two  of  the  sub-eorporate  approaehes  similar  to  lEEM  are  Eamily  of  Measures 
and  Cost-Based.  The  premise  of  the  Eamily  of  Measures,  like  that  of  lEEM,  is  that  no 
single  performanee  measure  ean  eneompass  the  range  of  work  undertaken  by  most 
business  proeesses.  A  range  of  measures,  therefore,  must  be  used  whieh  refleets  the  main 
inputs  and  outputs  of  the  department.  A  “family”  of  measures  attempts  to  measure  as  low 
as  possible  to  re  fleet  the  work  of  a  department  in  a  way  no  single  measure  ean.  Eor 
example,  the  key  performanee  measures  ehosen  for  a  management  aeeounts  department 
might  be: 

•  Reports  with  errors  /  Total  Errors 

•  Aetual  eost  /  Eoreeast  Cost 

•  Average  Report  Produetion  Time 

•  Reports  late  /  Total  Reports 

•  Number  of  reports  /  Number  of  staff 

•  Average  Cost  of  Reports 

These  measures  should  eneompass  the  range  of  inputs  used:  labor,  materials, 
eapital  and  equipment,  and  the  measures  should  eneompass  the  faetors  of  quantity, 
quality,  timeliness  and  eost.  A  Eamily  of  Measures  sueh  as  these,  related  to  the  real 
objeetives  of  a  department,  allows  the  level  of  performanee  to  be  measured  and 
monitored  over  time  to  establish  whether  it  is  stagnating,  deelining  or  improving.  It 
allows  the  effeet  of  produetivity  initiatives  to  be  established  and  foeuses  managers’ 
attention  on  the  need  to  improve  produetivity  —  a  true  and  proven  way  to  improve  a 
eompany's  eompetitive  position. 
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Unlike  the  lEEM  and  its  supporting  baseline  of  metries  and  eonversion  ratios,  the 
limitation  of  these  Eamily  of  Measures  approaches  is  that  they  do  not  provide  a  common 
theoretical  framework  and  consequent  common  unit  of  analysis  that  would  tie 
investments  in  IT  unambiguously  to  a  firm’s  economic  performance  (Bharadwaj  2000). 
The  lack  of  a  common  theoretical  framework  leads  to  an  inherent  problem  of  subjectivity. 
Though  subjectivity  is  not  completely  eliminated  in  the  lEEM,  analysts  and  researchers 
of  the  current  Eamily  of  Measures  approaches  are  left  to  develop  subjective  assessments 
of  how  the  variables  relate  to  each  other  rather  than  the  guidance  of  a  formal  framework 
or  mathematical  model.  In  most  cases,  simple  normalization  to  an  interval  scale  is  used 
along  with  an  assumption  of  linearity  of  the  relationship  among  the  variables  with 
weightings  provided  by  the  subjective  judgments  of  management  about  the  importance  of 
the  various  measures  (Housel  et  ah,  2001,  p.9). 

2,  Cost-Based  Similarities  and  Differences 

Although  there  is  also  a  cost-benefit  analysis  involved  in  the  metrics  supporting 
the  lEEM,  the  generally  accepted  conceptual  limitation  of  the  cost-based  approaches  to 
generating  a  return  on  investment-type  performance  ratio  is  that  they  do  not  have  a 
surrogate  for  revenue  (Johnson  1992).  The  problem  of  using  this  method  for  evaluating 
the  value  added  by  IT,  is  the  fact  that  if  cost  (or  any  of  its  derivatives)  is  used  as  a 
surrogate  for  value,  then  all  the  information  is  contained  in  one  term  of  the  ratio,  i.e.,  the 
denominator.  The  data  source  for  value  should  come  from  the  revenue  side  of  the  firm's 
performance  (i.e.,  numerator)  and  the  data  source  for  cost  (i.e.,  cost)  should  come  from 
the  cost  to  produce  the  firm's  outputs  (Housel  et  ah,  2001,  p.lO).  The  applicability  of  one 
popular  cost-based  technique,  known  as  Activity  Based  Costing  (ABC),  to  determne  ROI 
for  the  lEEM  and  its  baseline  of  metrics  and  conversion  rations  is  examined  and 
explained  in  greater  detail  in  Section  V.E.2:  Surrogate  for  Value. 


19 


D,  SUMMARY 

Intranet  portals  are  evolving  out  of  neeessity  to  better  manage  data  and  more  fully 
meet  the  needs  of  its  audienee:  the  enterprise  BDMs,  portal  owners  and  managers  and 
knowledge  workers.  Intranets  are  built  to  promote  effeetiveness  and  productivity  that 
indirectly  generate  more  precision  and  revenue.  Modifications  to  intranet  portals  and 
how  to  measure  their  efficacy  are  not  exclusively  about  the  immediate  and  direct  return 
of  dollars  in  and  of  themselves  as  many  ROIs  imply.  The  key  is  to  optimize  the  portal 
infrastructure  that  already  exists  and  the  processes  that  take  place  therein  with  minimal 
costs.  The  following  lEEM  and  supporting  metrics  provide  the  holistic  framework  and 
means  in  which  to  achieve  greater  optimization  through  comprehensive  and  logically 
model-based  quantitative  and  qualitative  cost-benefits  analyses  which  facilitate  seeking, 
interpreting  and  gaining  additional  value.. 
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III.  CONCEPTUAL  FRAMEWORK 


Everything  should  be  made  as  simple  as  possible,  but  not  simpler. 

Albert  Einstein 

A,  GENERAL  OUTLINE  OF  APPROACH 

To  better  appreciate  why  portals  exist  and  are  occasionally  re-designed,  one 
should  first  understand  the  different  domains  that  support  the  management  of  making 
information  findable  and  understandable,  their  distinct  constituents  and  how  these  are 
segmented  to  sustain  key  business  requirements.  The  lEEM  and  its  underlying  baseline 
of  metrics  were  determined  by  a  series  of  model/diagram  procedures  that  were  used  to 
break  down  the  composition  of  an  intranet.  In  order  to  reveal  an  abstract  domain 
analysis  view  and  the  association  therein  linking  metrics  groups  to  critical  business 
requirements,  an  affinity  diagram  is  used  to  create  the  conceptual  model  which  separates 
the  intranet  into  distinct  and  unique  segments  in  order  to  help  map  out  the 
problem/solution  space  analysis  (see  Eigure  2,  Intranet  Domains  and  Segments).  On  top 
of  this  diagram  the  various  users  and  their  roles  are  identified  within  each  segment  (see 
Eigure  3,  Constituent  Distribution  in  Segments  and  Teams  Responsible).  Then  an 
interrelationship  diagram  is  substituted  on  top  of  the  affinity  diagram  to  highlight  where 
pertinent  metrics  and  their  logical  relationship  between  related  users  and  their  roles  exist 
(see  Eigure  4,  IDEM  Outline  of  Metric  Types  and  Examples  in  Each  Segment).  These 
metrics  are  further  broken  down  into  hard,  soft  and  derived  forms  and  are  also  outlined 
on  the  diagrams  as  well  as  put  into  a  cause  and  effect  tree  table  (see  Eigure  7,  Ranking  of 
Key  Intranet  Metrics  by  Segment)  which  shows  the  impact  of  these  metrics  to  users  as 
well  as  strategic  business  values.  Easily,  the  table  is  put  into  a  prioritization  matrix  (see 
Table  2,  Example  of  One  Metric  Breakdown,  and  for  a  fuller  breakdown  Appendix  D, 
2 

Domain  analysis  is  a  systems  analysis  for  multiple  related  systems.  There  is  no  standard  definition  of 
domain  analysis;  several  domain  analysis  methods  exist.  Common  themes  among  the  methods  inelude: 

•  meehanisms  to  define  the  basie  eoneepts  (boundary,  seope,  and  voeabulary)  of  the  domain 
that  ean  be  used  to  generate  a  domain  arehiteeture  to  deseribe  the  data  (e.g.,  variables,  eonstants)  that 
support  the  funetions  and  state  of  the  system  or  family  of  systems. 

•  identify  relationships  and  eonstraints  among  the  eoneepts,  data,  and  funetions  within  the 
domain  identify,  evaluate,  and  seleet  assets  for  (re-)use  develop  adaptable  arehiteetures. 
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Intranet  Portal  Metric  Breakdown)  to  illustrate  levels  of  importance  and  to  establish  a 
baseline  of  metrics  in  which  to  commence  with  measurements.  Appendix  D  is  the 
current  proposed  baseline  set  of  metrics  and  conversion  ratios  that  result  from  high-level 
analysis  of  intranet  efficiency  and  effectiveness  factors  that  gauge  and  achieve  greater 
value  and  value-creating  benefits. 

The  lEEM  focuses  on  strategic  fit,  functional  objectives  and  the  opportunity  or 
necessity  for  making  process  improvements  as  the  keys  to  success.  In  addition,  the  lEEM 
introduces  a  common  theoretical  framework  that  has  been  missing  in  which  to  measure 
ah  important  facets  of  intranet  processes  critical  to  assessing  value.  Nevertheless,  its 
holistic  approach  does  not  eliminate  subjectivity  altogether  as  it  accounts  for  critical 
qualitative  factors  which  are  often,  if  not  entirely,  overlooked  in  other  commonly  used 
measuring  techniques  which  concentrate  on  usage  statistics,  such  as  traffic  volume.  In 
short,  as  a  result  of  being  a  more  comprehensive  model  some  soft  metrics  must  be  taken 
into  account  which  are  not  strictly  quantitative  or  without  human  interpretation  or 
assumption.  Nevertheless,  when  parties  agree  upon  a  relatively  well-defined  set  of 
performance  metrics,  it  is  possible  to  have  a  relatively  unambiguous  collaborative 
interpretation  of  the  phenomenon  (Baskervihe,  1999,  pp.  12).  In  addition,  many  of  these 
soft  metrics  can  be  interpreted  quantifiably  (see  Section  IV.B.2:  Periodic  Soft  Metrics). 

Invariably,  numerical  constructs  and  identifying  assumptions  regarding  cost 
savings  or  capital  investment  and  revenue  generation  are  problematic.  There  is  no  hard 
and  fast  rule  for  defining  these  assumptions;  however,  a  comprehensive  model  that 
stresses  strategic,  functional  and  technical  fit,  and  seeks  to  identify  opportunities  for 
process  improvements,  is  more  likely  to  result  in  top  management  support  because  it  is 
based  on  their  direction  for  the  organization,  not  merely  the  merits  of  a  particular 
technology.  Along  these  lines,  lEEM  is  particularly  well  suited  to  concurrence  because  it 
systematically  takes  into  account  user  behavior  and  feedback  which  are  critical  facets  to 
understanding  and  optimizing  processes  which  propel  intranet  utility. 
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1,  Domains  and  Segments  Underlying  the  Intranets 

The  segments  outlined  below  represent  six  perspeetives  of  intranet  effeetiveness 
measurement,  all  of  whieh  have  an  underlying  dependency  on  each  other.  Efficiency  and 
routing  provide  the  basics  that  must  be  maximized  before  content  and  design  can  become 
effective.  Content  and  design  may  then  contribute  to  overall  channel  effectiveness  which, 
in  combination  with  other  channels,  provides  a  healthy  bottom  line;  meeting  the  business 
requirements.  Corporations  should  take  the  underlying  dependency  into  account,  to  make 
sure  there  is  an  effective  growing  path  as  they  phase  in  their  intranet  analytics. 


Front-End  Ensure  Intuitive  Navigation 

Quaiity,  Authoritativeness,  Personaiized 


Back-End 

Develop  Shared 
Taxonomy,  Global  Search 
Across  Critical  Content 
Domain  Integration 


Design 


Content 


Business 

Requirement 


Domain 

Infrastructure 


People,  Process,  Technology 


Figure  2,  Intranet  Domains  and  Segments 


^  The  reason  for  the  delineation  and  the  differenee  between  effieieney  and  effeetiveness  shown  above  is 
addressed  at  the  end  of  Seetion  III.B,  Hard,  Soft  and  Derived  Metrics  for  the  Intranet,  and  again  in  eontext 
to  Return  on  Investment  eonsiderations  in  section  V.D.2.a,  Quantitative  and  Qualitative. 
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Web  analytics  are  a  vital  part  of  managing  any  Web  portal,  but  the  terminology, 
the  technology  and  its  use  are  far  from  being  mature.  A  major  obstacle  that  inhibits 
enterprise-level  valuations  of  virtually  any  kind  is  the  lack  of  consistent  data  definitions 
across  different  portals/groups.  Consequently,  the  lEEM  provides  a  comprehensive 
reference  of  this  sort  to  alleviate  uncertainty  and  facilitate  concurrence  (see  Appendix  E, 
Glossary,  for  a  comprehensive  and  analytical  set  of  terms  oriented  for  an  intranet). 
Similarly,  this  section  following  explains  the  model  parts  illustrated  in  Eigure  2  above: 

a.  Domains 

Segments  and  constituents  of  the  intranet  fall  into  one  of  three  domains: 
Eront-end,  Back-end  and  People,  Process  and  Technology: 

(1)  Front-End.  Eront-end  is  the  processes  in  which  a  knowledge 
worker  has  a  more  direct  interface,  i.e.,  user  experience  and  design  or  presentation  of 
content. 

(2)  Back-End.  Back-end  is  the  processes  that  take  place  that  a 
knowledge  worker  does  not  see,  but  can  be  objectively  measured,  i.e.,  by  directly 
interpreting  server  log-file  data,  packets  or  server  requests. 

(3)  People  Process  and  Technology.  This  domain  is  where 
challenges  of  people,  processes  and  technology  meet.  Eor  an  enterprise  intranet  its 
purpose  is  to  develop  an  improved  information  finding  capability  by  providing  an  intranet 
environment  that  consistently  supports  both  browse  and  search  so  that  knowledge 
workers  can  confidently  and  seamlessly  find  the  technical  information  they  need  to  work 
effectively  and  efficiently,  thereby  increasing  value. 

b.  Front-End  Segments 

The  Eront-end  domain  can  be  further  segmented  into  two  underlying  areas 
that  knowledge  workers  directly  encounter: 

(1)  Design  -  Editorial  Programming.  The  editor’s  experience 
should  be  a  straightforward  and  consistent  use  of  accepted  best  practices  and  best  tools 
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available  (approved  by  the  company)  to  organize,  access  and  re-use  information  in  a 
desired  manner  to  produce  a  desired  outcome. 

(2)  User  Experience.  The  relationship  and  cognitive  connection 
knowledge  worker  has  with  the  intranet  and  the  information  provided. 

c.  Back-End  Segments 

The  Back-end  domain  can  be  further  segmented  into  three  underlying 
areas  that  knowledge  workers  don’t  directly  come  across  while  using  an  intranet: 

(1)  Usage.  Technical  information  that  relates  to  or  specializes  in 
techniques  or  subjects  of  applied  science.  Examples  of  this  include  numbers,  time  and 
duration  along  with  userlD  and  other  work  related  information,  i.e.,  department,  location 
and  position. 

(2)  Content.  All  documents  —  help  text,  search  results,  forms,  or 
application  information  —  that  is  delivered  to  a  knowledge  worker  via  a  website,  or 
application.  It  is  both  subject  and  specific  information.  For  example,  “People”  is  a 
subject  and  a  person  by  name  is  specific  information.  Content  is  organized  and  found 
depending  on  its  properties  and  associations.  Being  a  subjective  or  soft  metric 
measurement,  the  effectiveness  of  content  is  not  something  that  can  be  evaluated  using 
straightforward  clickstream  analysis  (number  of  clicks  to  discovering  information 
sought).  It  is  not  directly  quantifiable  and  requires  deeper  insight  into  how  the  content  of 
the  website  is  perceived,  for  example,  through  the  use  of  surveys. 

(3)  Domain  Infrastructure.  This  is  both  the  physical  and  software 
interaction  representation  of  the  relationships  between  the  key  elements  within  a  design. 
It  shows  how  these  elements  interact  and  must  “transfer”  information,  so  that  the 
architect  can  build  the  environment.  Examples  are  information  maps  (document 
elements),  server  topography,  scope  of  services,  permissions  (ownership). 
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d.  Business  Requirements  Segment 

The  focus  of  this  segment  is  on  the  outcome  of  processes  which  enable 
value.  lEEM  regards  process  as  the  most  critical  enabler  as  improvements  to  it  harness 
the  merits  of  technology  and  provide  back  the  greatest  ROI  and  benefits  to  people. 

(1)  Business  Requirements.  The  over-arching  goal  of  grouping 
metrics  within  and  across  segments  is  to  track  how  well  business  requirements  are  being 
supported  which  promote  productivity.  Although  it  is  in  a  separate  domain,  all  segments 
and  their  constituents  from  the  Front  and  Back-end  domains  are  designed  to  support 
critical  business  requirements  (these  requirements  are  outlined  in  section  III.E,  Critical 
Business  Requirements). 

2,  Constituents 

Constituents  represent  the  data  necessary  in  the  Front  and  Back-end  domains  to 
find  information  in  support  of  the  third  domain  of  People,  Process,  and  Technology, 
specifically  those  requirements  outlined  in  section  III.E,  Critical  Business  Requirements. 
The  problem  portals  try  to  overcome  is  the  same  as  the  main  reason  they  are  created:  to 
facilitate  knowledge  workers’  discovery  of  all  the  information  they  need  to  do  their  jobs 
better.  This  then  begs  the  question  of:  “What  makes  information  more  discoverable?”  In 
order  for  information  to  be  found  and  useful,  portal  design  must  sustain  the  following 
five  vital  requisites: 

•  Amount  of  information  and  meta-information  (accessibility  &  manageability) 

•  Understanding  information  seeking  and  use  behavior 

•  Navigation  system  design  used  to  expose  information  (logically  grouped) 

•  Confidence  in  quality  of  information  and  meta-information 

•  Relevancy  to  knowledge  worker 

Thus  any  portal  desiring  to  be  part  of  the  information  system  should  contain  the 
following  13  constituents  of  discovering  information  that  collectively  sustain  the 
requisites  above.  Eor  the  sake  of  simplicity  and  logic,  six  of  these  constituents  are 
grouped  below  into  the  Back-end  as  are  the  remaining  seven  into  the  Eront-End  domains 
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(examples  and  a  fuller  explanation  of  eaeh  eonstituent  ean  be  found  in  Appendix  A, 
Constituents  to  Finding  Intranet  Information): 

a.  Back-End  Constituents 

(1)  Content  Properties.  An  integral  eonneetion  to  eontent  is  its 
eontent  properties.  How  eontent  is  deseribed  in  order  to  affeet  the  assoeiations  made  in 
information  retrieval  and  presentation  is  done  through  the  eontent  properties  tagged  to  it. 
The  eharaeteristies  of  a  eontent  item  make  up  its  properties  (a.k.a.  attributes),  sueh  as 
author,  length,  name,  ete.,  ean  be  represented  with  a  sehema  and  supported  by 
voeabularies  of  metadata  (see  Appendix  A). 

(2)  Domain  Information  Infrastructure.  The  domain  information 
infrastrueture  (DII)  of  a  eorporation  is  the  sum  and  organization  of  all  its  data, 
taxonomies,  tools  and  produets.  DII  for  Web  analyties  needs  to  eneompass  not  all  the 
information  arehiteeture  that  is  available  but  only  the  best  elements  of  these  groups  that 
ean  be  further  developed  and  integrated  to  improve  eontrol  of  eontent  and  eontext  to  meet 
knowledge  workers  needs  and  to  exeeed  their  expeetations  as  well 

(3)  Domain  Integration  Framework.  This  framework  is  the  virtual 
representation  of  the  relationships/strueture  between  the  key  elements  within  a  design.  It 
shows  how  these  elements  internet  and  must  “transfer”  information,  so  that  the  arehiteet 
ean  build  the  environment.  Examples  are  information  maps  (doeument  elements),  server 
topography,  seope  of  serviees,  permissions  (ownership). 

(4)  Information  Life  Cycle.  This  eyele  refers  to  events  often 
repeated  again  and  again  in  maintaining  the  relevanee  and  aeeessibility  of  eontent  in  an 
information  system.  These  events  inelude  updating,  versioning,  arehiving  and,  when 
neeessary,  deleting. 

(5)  Search.  Contrary  to  other  eommon  notions  of  seareh,  within 
the  lEEM  it  refers  to  an  application  employed  by  knowledge  workers  as  a  tool  to  find 
through  direet  surfaeing  or  through  surfaeing  an  obvious  navigational  path. 
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(6)  User  Data.  Data  comprised  of  facts  and  figures  a  knowledge 
worker  maintains  private  aceess  to  for  knowledge  retention  and  expansion.  This  is  also 
referred  to  as  “personalization”. 

b.  Front-End  Constituents 

(1)  Accessibility.  Information  is  eonsidered  aceessible  when  it  is 
available,  reaehable,  and  understandable.  (Assuming  the  information  exists,  ideally 
availability  could  occur  from  any  point  of  access  of  and  point  of  entry  to  the  information 
system.  Although  how  reaehable  information  is  depends  on  a  number  of  factors;  in  this 
context  the  more  information  is  eompatible  with  a  shared  Information  Architecture,  the 
easier  it  is  to  retrieve.  How  this  information  is  then  packaged  and  presented  in  a  fashion 
that  can  be  understood  by  the  knowledge  worker  is  the  last  stage  of  accessibility.) 

(2)  Communication  of  Authoritativeness  and  Importance. 
Communication  of  authoritativeness  and  importanee  is  aehieved  by  communieating  to  the 
knowledge  worker  the  credibility  of  an  information  system  within  that  system  to  ensure 
confidence  and  trust  in  it,  i.e.,  by  showing  that  information  comes  from  a  respeeted  and 
newsworthy  source  within  the  organization  or  outside,  such  as  indicating  that  the  article 
is  from  the  Human  Resouree  Department  or  from  a  well  known  national  syndicate  news 
organization.  Another  example  of  this  is  an  organization  intranet  devoted  strictly  to 
business  related  websites. 

(3)  Communication  of  Understanding  Search.  Communication  of 
understanding  search  is  achieved  by  communieating  to  the  knowledge  worker  meaning 
and  signifieance  of  the  information  they  are  viewing  by  keeping  it  eonsistent  to  ensure 
acceptance  and  engagement,  whieh  further  inereases  the  audienee’s  propensity  to  act  on 
information  in  a  common  way,  i.e.,  a  Glossary  of  Terms  and  Definitions,  Mouse-over, 
and  best  practices  that  are  eonsistently  applied  to  all  intranet  websites. 

(4)  Information  Grouping  and  Segmentation.  Information 
grouping  is  the  logieal  collection  of  relevant  and  similar  information,  such  as  Content 
Nuggets  and  Categories  respectively.  Another  aspeet  of  this  is  information  segmenting  in 
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which  only  the  relevant  parts  of  a  doeument,  i.e.  a  paragraph  or  two  are  extraeted  and 
placed  into  fragmented  groupings  instead  of  the  entire  doeument.  For  example,  first  200 
words  of  a  doeument  returned  as  part  of  a  search  result. 

(5)  Navigation  -  Local  and  Global.  Navigation  is  a  method  of 
moving  through  the  domain  framework  by  way  visual  presentation  and  eonsistent 
ehoiees.  Navigation  has  2  basie  types:  loeal  (also  known  as  vertieal)  and  global  (also 
known  as  horizontal): 

•  Local  navisation  presents  ehoiees  leading  to  subtopies  or  sub-areas  of  a  site, 
usually  defined  by  one  of  its  main  menu  subjeets  (often  referred  to  as  a  “drill  down”). 

•  Global  navisation  presents  ehoiees  leading  to  other  main  areas  of  a  site,  i.e. 
Home;  Seareh;  About.  Global  navigation  is  eonsistent  throughout  an  information  system 
whieh  allows  knowledge  workers  to  go  aeross  portals  with  a  sense  of  familiarity. 

(6)  Personalization.  A  method  of  eontextualizing  information  for  a 
knowledge  worker  based  on  what  is  known  about  the  knowledge  worker.  A  site  ean  use 
personalization  to  alter  navigation  and  eontent  presented  aeeording  to  the  pereeived  needs 
of  the  knowledge  worker.  Examples  include  eontent  filtering  based  on  role; 
authentieation  based  on  name  or  status  as  a  manager. 

(7)  User  Assistance.  Help  made  available  to  the  knowledge  worker 
while  using  an  information  system.  User  assistance  provides  guidanee  on  how  to  use  the 
system  or  additional  help  in  finding  information  sought.  Examples  are  help  in  eontext, 
feedback.  Dialog  boxes,  and  training,  real  time  assistanee  from  a  human  intermediary  via 
a  live  ehat. 


The  signifieanee  in  Eigure  3  below  is  the  majority  of  constituents 
(seven)  required  in  sustaining  the  five  vital  requisites  to  find  useful  information  fall  into 
the  Front-end  domain.  This  is  erueial  in  understanding  that  the  metries  taken  from  this 
end  of  the  intranet  must  be  taken  into  eonsideration  if  portals  are  to  be  measured  more 
eompletely  with  better  balanee  than  they  are  today.  Eailure  to  do  this  results  in  less 
“impactful”  actions  taken  to  inerease  value. 
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Front-End  Ensure  Intuitive  Navigation  Quality, 
Authoritativeness,  Personalized  Experience 

Accessibility 
Navigation 
Customization/ 
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Figure  3,  Constituents  Distribution  in  Segments  and  Teams  Responsible 

B,  HARD,  SOFT  AND  DERIVED  METRICS  FOR  THE  INTRANET 

To  measure  efficacy  of  a  corporate  intranet’s  portals  requires  a  coherent  and 
balanced  combination  of  metrics  taken  from  all  segments.  When  these  metrics  are 
collectively  and  uniformly  applied  in  periodic  measurements,  they  can  indicate  tractable 
improvements  over  time.  In  order  to  do  this,  these  metrics  should  be  grouped  into  metric 
categories  that  support  business  requirements  (see  Appendix  C,  Example  of  Metrics  to 
Measure  Intranet  Performance,  and  Appendix  D,  Intranet  Portal  Metric  Breakdown).  To 
this  end,  there  are  essentially  three  types  of  metrics:  hard,  soft  and  derived. 

•  Hard  metrics  can  be  objectively  measured,  i.e.,  by  directly  interpreting  server 
log-file  data,  server  requests,  number  of  visitors  over  a  given  period  of  time,  etc. 
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•  Soft  metrics  involve  many  subjective  and  qualitative  aspects  that  provide  a 
frame  of  reference  to  interpret  the  results,  i.e.,  survey  results,  visual  analysis  and 
usability. 

•  Derived  metrics  consist  of  hard  and  soft  metrics  from  a  variety  of  business 
and  knowledge  data  involved  and  a  subsequent  educated  assumption  to  draw  conclusions, 
i.e.,  estimates  of  speed  to  market,  value,  loyalty  and  reach. 

With  respect  to  the  preceding  discussion  and  the  introduction  of  these  metric 
types,  a  theoretical  model  (see  below  Figure  4,  IDEM  Outline  of  Metric  Types  and 
Examples  in  Each  Segment)  can  be  created  illustrating  the  placement  of  domains, 
segments,  constituents  as  well  as  a  sampling  of  metric  groups  that  lead  to  deriving  the 
successfulness  of  portals  in  supporting  business  requirements.  A  distinction  can  now  also 
be  made  that  separates  efficiency  from  effectiveness:  efficiency  is  measured  with 
predominantly  quantifiable  or  hard  metrics,  i.e.,  numbers  and  durations  of  time  or  both; 
effectiveness  is  this  and  more  as  it  takes  into  account  qualitative  factors. 
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Front-End  Ensure  Intuitive  Navigation  Quality, 
Authoritativeness,  Personalized  Experience 


Accessibility 
Navigation 
Customization/ 
Personalization 
User  Assistance 
User  Data 
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People,  Process,  Technology 

Challenge  and  Vision:  Given  knowledge  workers  can't  find  all  information 
needed,  esp.  technical  info  and  experts,  with  portals  that  lack  consistency  in 
management  and  few  central  management  services  in  an  enterprise  intranet 
that  is  afflicted  with  duplication/  redundancies  and  no  business  intelligence, 
develop  an  improved  information  finding  capability  by  providing  an  intranet 
environment  that  consistently  supports  both  browse  and  search  so  that 
knowledge  workers  can  confidently  and  seamlessly  find  the  technical 
information  they  need  to  work  effectively  and  efficiently. 


Figure  4,  lEEM  Outline  of  Metric  Types  and  Examples  in  Each  Segment 


C.  AUDIENCES 

An  intranet  services  three  distinct  audiences:  Enterprise  business  decision 
makers,  Portal  owners  and  managers,  and  Users  (or  knowledge  workers).  The  interests 
of  all  three  of  these  audiences  must  be  taken  into  account  and  distinguished  as  much  as 
possible  to  better  organize,  prioritize  and  conduct  metrics  to  measure  effectiveness  of  an 
intranet.  If  people  are  to  be  given  benefits  from  enhancements  to  the  enabler  of  process, 
then  it  is  essential  to  understand  who  and  where  they  are  as  well  as  the  roles  they  play  in 
fulfilling  business  requirements. 


1,  Enterprise  BDMs 

Enterprise  is  the  entire  operation  of  a  commercially  organized  company  whose 
business  activities  are  aimed  specifically  at  growth  and  profit  through  the  making  of 
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products,  buying  and  selling  of  goods,  and/or  providing  services.  The  decisions  made  by 
high  level  executives  based  off  what  they  or  their  staffs  discover  in  the  intranet  can  have 
a  far  reaching  effect  on  the  other  divisions  and  departments  throughout  the  organization. 
Enterprise  BDMs  are  inevitably  responsible  for  ensuring  the  company  meets  its  business 
requirements,  therefore  providing  insightful  measurements  to  how  the  intranet  is  driving 
value  back  into  the  company  is  a  litmus  test  of  great  importance  to  them. 

2,  Enterprise  Portal  Owners  and  Managers 

Portals  are  a  means  into  an  information  system,  such  as  a  corporate  intranet, 
established  over  a  coherent  body  of  information  or  community  of  interest  (a.k.a.  a  site)  to 
provide  more  personalized  and  relevant  information  to  its  users.  Examples  in  an 
organization  are  a  Einance  website  or  HR/Personnel  website.  Portal  owners  are  usually 
the  same  as  the  director  or  general  manager  of  a  large  group  who  directs  the  creation  of 
the  group’s  own  portal  to  better  support  its  own  operations  as  well  as  service  others.  The 
more  the  information  in  their  portal  is  leveraged  by  other  portals  and  users,  the  more 
valuable  their  portal  becomes  to  the  company  -  a  distinction  for  which  owners  strive. 
Portal  managers  maintain  the  portals  and  are  responsible  for  helping  the  business  owner 
meet  business  requirements  through  the  optimization  of  the  portal  itself.  Measuring  how 
the  portal  is  used  in  order  to  improve  its  performance  in  this  vane  is  of  peak  interest  to 
both  groups  of  managers,  with  the  preponderance  of  technical  work  left  to  the  “hands-on” 
portal  manager. 


3,  User  or  Knowledge  Worker 

A  user  is  one  who  gathers,  analyzes,  adds  value  and  communicates  information  to 
empower  decision-making.  The  nature  of  ‘k-work’  is  ad  hoc,  demand-driven  and 
creative,  both  in  the  ability  to  create  new  knowledge  greater  than  the  sum  of  its  parts  and 
in  the  ability  to  present  the  knowledge  in  a  highly  communicative  ways  (Mattews,  2000, 
p.  39).  A  critical  and  mostly  overlooked  aspect  in  deriving  and  prioritizing  more 
comprehensive  sets  of  metrics  needed  to  gauge  effectiveness  is  a  deeper  study  of  the  user 
experience  along  with  more  emphasis  on  specifically  what  users  need  in  order  to  do  their 
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jobs  better,  i.e.,  do  users  seek  technical  information  —  and  if  so  —  is  it  code  or  white 
papers,  etc.  In  this  way,  portal  optimization  will  gradually  provide  knowledge  workers 
what  they  need  as  well  as  eventually  giving  them  what  they  didn’t  know  they  needed  (as 
content  management  and  personalization  techniques  mature  and  become  more 
sophisticated). 

D,  RELATIONSHIPS  OF  DOMAINS,  SEGMENTS,  AUDIENCES  &  METRICS 

When  the  model  is  viewed  to  see  how  the  domains  break  out  with  respect  to 
audiences  and  metrics,  the  following  conclusions  stand  to  reason;  Where  users  interact 
with  metrics  directly  occurs  in  the  Front-end.  The  majority  of  these  metrics  are  “soft”  in 
nature.  Examples  of  popular  soft  metrics  in  the  Design  segment  of  the  Front-end  include 
visual  analysis,  usability  and  ease  of  navigation.  However,  there  are  a  number  of  hard 
metrics  which  can  be  derived  to  better  account  for  the  User  Experience  segment,  such  as 
number  of  mouse  clicks  to  find,  time  to  locate  information  sought  and  page  abandons. 
Where  portal  owners  are  most  interested  in  and  what  portal  managers  most  interact  with 
are  metrics  in  the  back-end  domain.  The  majority  of  these  metrics  are  “hard”  in  nature. 
Examples  of  popular  hard  metrics  include  top  downloads,  unique  users  and  duration  on 
site.  However,  there  are  a  number  of  soft-related  metrics  in  this  domain,  particularly  in 
the  Content  segment,  such  as  surveys  and  best  bets.  Enterprise  BDMs  have  a  keen 
interest  in  the  remaining  segment.  Business  Requirements.  These  requirements  are  vital 
to  the  productivity  of  the  company  and  are  what  all  other  segments  from  inception  are 
designed  to  support  and,  by  extension,  what  the  metrics  associated  to  them  as  a  result  of 
the  lEEM  should  ultimately  gauge,  i.e.,  reach,  loyalty  and  value.  Figure  5  below 
highlights  which  metrics  groups  and  types  fall  under  each  audience  as  well  as  how  and 
where  each  corresponds  to  the  Intranet  Efficiency  and  Effectiveness  Model. 
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METRICS 


Figure  5,  Overlay  of  Domains,  Audiences  and  Metrics  to  Segments 

*  Usage  segment  metries  represent  majority  metries  used  to  determine  portal  performance. 


After  the  variety  of  Web  metrics  are  tabulated  (see  Appendix  C:  Metrics  to 
Measure  Intranet  Performance),  there  are  approximately  30  times  as  many  metrics  in  the 
Usage  segment  as  there  are  in  the  other  segments.  Moreover,  these  metrics  are  all  hard 
metrics.  The  fact  that  engineers  or  people  with  technical  backgrounds,  who  generally 
conduct  metrics  in  organizations  in  the  first  place,  tend  to  favor  quantifiable  metrics 
along  with  their  simplicity  and  availability  often  results  in  an  unbalanced  approach  to 
measurement,  i.e.,  not  taking  in  periodic  and  systematic  soft  metrics,  particularly  those 
that  most  impact  users  from  the  users  perspective.  Figure  6  below  illustrates  the  relative 
number  of  metrics  commonly  used  within  each  segment. 
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lEEM  Metric  Proposition: 

Given  there  are  few  metrics  available  in 
other  segments  outside  Usage  and  given  that 
part  of  the  premise  of  lEEM  success  is  that 
finding  information  is  more  effective  when 
Font-end  factors  are  better  accounted,  then  a 
more  coherent  selection  of  metrics  from 
Usage  AND  from  other  segments  needs  to 
be  created  which  take  into  fuller  account 
user  behavior  and  are  delineated  between 
efficiency  and  effectiveness  for  each 
audience  (enterprise,  portal  and  user)  to 
provide  more  consistent  and  meaningful 
measurements  that  indicate  improvement 
gain  quantifiably,  qualitatively 
and  anecdotally. 


METRICS 


Enterprise 


Portals 


Users 


Figure  6,  Intranet  Segment  Metrics  Number  Comparison 

*Ratio  of  Usage  metrics  is  30+  to  1  over  all  other  metrics  from  all  other  segment  areas. 


Many  corporations  today  focus  on  routing  metrics,  such  as  number  of  hits  per 

page,  top  10  search  strings,  most  popular  downloads  and  number  of  referrals  from  other 

sources  (such  as  banner  advertising,  search  engines  and  direct  links).  These  metrics  are 

popular  and  have  been  created  in  far  greater  numbers  because  they  address  the  issues 

many  departments  and  groups  are  facing  today,  namely  accessibility  and  visibility  of 

their  website.  These  usage-related  metrics  are  also  popular  because  they  can  be  done 

quickly  and  are  more  mathematically  straightforward,  favoring  the  proclivity  of 

technically  oriented  workers  who  usually  do  the  measuring,  than  the  more  time 

consuming  soft  metrics  concerned  with  such  things  as  user  behavior  and  experience  in 

the  Front-end.  In  addition,  routing  metrics  are  relatively  simple  to  understand  at  the 
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business  level  and  the  data  is  relatively  easy  to  eollect  using  the  Web  server  log  files.  In 
fact,  most  Web  analytics  packages  provide  many  routing  metrics  as  pre-packaged  reports, 
so  it  is  natural  to  defer  to  these  “out-of-the-box”  tools.  Unfortunately,  since  they  are 
often  used  for  Internet  websites  they  are  mistakenly  applied  in  like  fashion  to  corporate 
intranets  and  in  a  lopsided  manner. 

The  two  audiences  with  different  needs  that  are  driving  portal  software 
development  are  the  portal  managers  who  want  a  centralized  framework  for  integrating 
and  exposing  line-of-business  applications,  and  BDMs  who  want  to  make  their 
knowledge  workers  more  productive  at  sharing  information  and  working  together.  In 
short,  IT  wants  plumbing  and  BDMs  want  productivity.  However,  simple  statistics  alone 
on  plumbing  are  not  going  to  provide  BDMs  with  all  feedback  needed  to  track 
improvements  towards  productivity.  Oddly  enough  the  audience  most  overlooked  in 
ascertaining  intranet  performance  is  the  user,  despite  the  fact  that  many  of  the 
constituents  necessary  to  sustain  the  requisites  of  finding  information  are  found  in  the 
Front-end  domain  where  the  knowledge  worker  resides.  Consequently,  when  selecting 
metrics  to  measure  intranet  effectiveness,  due  consideration  needs  to  be  given  to  metrics 
in  the  Design  and  User  Experience  segments. 

E.  CRITICAL  BUSINESS  REQUIREMENTS 

By  determining  from  all  segments  which  complementary  metrics  can  be  coupled 
together  in  groups  and  which  groups  collectively  best  indicate  how  well  a  portal  supports 
a  business  requirement,  efficacy  indications  are  attained.  It  is  through  the  refinement  of 
these  combinations  and  groupings  of  metrics  (each  organization  is  unique  and  therefore 
should  take  steps  to  refine  their  metrics  after  periodic  measurements)  from  all  segments 
of  the  Intranet  that  leads  to  improvement  in  critical  business  requirements.  The  creation 
of  these  metrics  requires  a  multidisciplinary  approach  and  overview.  A  good  baseline  for 
these  metrics  and  their  conversion  ratios  is  outlined  in  Appendix  D,  Intranet  Portal 
Metric  Breakdown,  and  the  composition  and  manner  a  multidisciplinary  team  should 
administer  and  steward  these  metrics  is  presented  in  Chapter  VI,  The  Business 
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Intelligence  Team.  To  this  end,  the  following  eritical  business  requirement  terms  are 
defined  with  regard  to  the  quality  paradigm’s  pursuit  for  greater  value  and  within  the 
eontext  of  what  enterprises  strive  for  to  be  productive: 

1.  Agility 

Agility  is  the  ability  both  physical  and  mental  to  react  nimbly  and  deftly  to 
change,  i.e.,  alert  combination  of  speed,  skill  and  intelligence  of  work  and  business 
transactions  to  maintain  a  competitive  edge.  Examples  of  this  include  the  ability  to  adjust 
to  the  information  needs  of  a  specific  audience  in  order  to  provide  them  more 
personalized  and  relevant  data  in  short  order.  This  can  be  presented  by  how  well  portal 
managers  analyze  user  behavior  data  taken  during  visits  to  sites  and  then  optimize  the 
sub-sites,  pages  and  links  accordingly. 

2.  Disintermediation 

Disintermediation  is  the  removal  of  parties  normally  involved  in  a  process,  i.e.,  a 
business  transaction  between  producers  and  consumers.  Examples  of  this  are  cross  portal 
referrals  that  allow  users  to  link  where  they  need  to  be  with  not  having  to  rely  on  others 
or  begin  making  phone  calls. 

3.  Loyalty 

Loyalty  is  feeling  and  acting  upon  a  sense  of  duty  out  of  an  attachment  to 
something  beneficial,  i.e.,  devotion  to  a  particular  site  that  provides  useful  information  to 
get  a  job  done  correctly  and  quickly.  This  can  be  indicated,  for  example,  by  the  number 
of  return  users  and  growth  of  new  users. 

4.  Opportunity 

Opportunity  is  an  advantageous  chance:  a  chance  brought  about  by  coincidence  or 
by  design  through  a  combination  of  favorable  circumstances.  Eor  example,  modifying 
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the  type  and  number  of  links  on  what  is  determined  to  be  the  most  visited  page  should 
present  an  opportunity  for  more  traffie  to  links  that  were  less  exposed  and  frequented. 

5.  Reach 

Reach  is  to  extend  the  range  of  influenee  as  far  as  a  partieular  teehnology  and 
serviee  allows  in  order  to  impaet  on  people  or  on  a  group,  i.e.,  a  portal  to  sehedule  and 
promote  training  wishes  to  toueh  as  many  potential  customers  within  a  company  as 
possible.  This  can  be  determined  in  part  by  monitoring  the  number  of  new  and  unique 
users  from  different  organizations  and  roles. 

6.  Return  on  Investment 

ROI  is  a  financial  ratio  measuring  the  cash  return  from  an  investment  relative  to 
its  cost.  Revenue  is  normally  recorded  when  the  product  or  service  is  delivered  or 
ownership  of  it  changes  to  the  customer.  Although  costs  can  be  associated  to  nearly 
every  business  function  (i.e.,  budgets),  costs  associated  with  portals  is  less  clear  cut  as 
well  as  the  revenue  it  generates.  Therefore,  Intranet  ROI  should  be  gauged  through 
indicators  of  effectiveness  that  support  the  business  requirements  designed  to  increase 
productivity.  Measuring  ROI  within  the  context  of  this  framework  is  derived  from 
analysis  taken  from  and  qualified  by  all  metric  sub-groups  within  each  portal  and 
aggregately  across  the  enterprise  (see  Chapter  V,  Return  on  Intranet  Metrics  Investment). 

7.  Value 

Value  in  this  context  is  an  adequate  or  satisfactory  return  on  or  recompense  for 
something  of  worth,  importance  and  usefulness,  i.e.,  how  valuable  is  the  functionality  and 
performance  of  a  portal.  Although  portals  impact  matters  of  monetary  worth,  their  value 
expressed  in  terms  of  money  is  of  lesser  consideration  compared  to  the  value  they  return 
to  an  enterprise  in  managing  knowledge  and  facilitating  productivity.  Similar  to  ROI, 
value  can  be  derived  by  reduced  time  constraints  and  increased  growth  in  current  and 
new  audiences. 
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8,  Creativity 

Creativity  is  the  ability  to  use  the  imagination  to  develop  new  and  original  ideas 
or  things,  i.e.,  a  new  produet  line  or  approaeh  to  solve  a  problem.  An  example  could  be 
more  cross  portal  referrals  in  which  users  of  one  portal  are  now  directed  to  more  relevant 
sources  of  information  in  other  portals  not  previously  known,  thereby  increasing  their 
chances  to  be  creative  with  this  wider  variety  of  data  along  with  the  subsequent  and 
synergistic  cognitive  connections  they  will  make  between  them. 

9.  Better  Decisions 

Better  Decisions  are  all  about  all  corporate  employees  making  better  informed 
decisions  after  considering  all  pertinent  information  and  choices  possible.  This  metric  is 
certainly  derived  and  anecdotal  and  is  essentially  a  consideration  process  of  a 
combination  and  culmination  of  derived  metrics  which  gauge  the  direction  of  business 
requirement  factors  (see  Figure  10,  Improving  and  Reducing  the  Decision  Points). 

F.  SUMMARY 

In  effect,  the  inter-disciplinary  derivation  of  the  lEEM  represents  a  form  of  data 
abstraction  at  multiple  levels,  making  it  an  art  as  well  as  a  science.  It  is  representative  of 
why  software  engineers  who  are  tasked  to  do  this  type  of  work  benefit  significantly  more 
from  multi-disciplinary  experiences  than  they  would  from  predominately  one 
background.  This  holistic  approach  is  essential  to  determining  all  segments  and  their  role 
in  the  human  information  processes  within  an  intranet.  Eor  example,  defining  a  business 
by  the  number  of  "customers"  it  possesses  is  meaningless  because  loyalty  continues  to 
elude  many  sites.  Thus,  businesses  must  evaluate  themselves  with  metrics  that  provide 
meaningful  insight  into  how  well  they  execute  their  critical  business  requirements,  not 
merely  how  well  they  are  able  to  drive  visitors  and  elicit  sporadic  transactions.  The 
lEEM  framework  helps  fulfill  this  need.  Although  no  single  tool  addresses  all 
perspectives  equally  well  in  an  efficient  manner,  corporations  need  to  realize  that 
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measuring  the  effectiveness  of  activity  on  intranet  portals  requires  periodic  application  of 
multiple  tools  and  techniques.  How  to  apply  the  lEEM  in  light  of  this  is  addressed  next 
in  Chapter  IV,  Applying  the  lEEM  Framework  to  Key  Business  Requirements . 
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IV.  APPLYING  lEEM  FRAMEWORK  TO  KEY  BUSINESS 

REQUIREMENTS 

Most  of  our  intranet  is  about  top-line  growth,  not  the  cost  reduction 
associated  with  ROI.  Intranet  applications  often  provide  new  ways  of  working  - 
and  how  do  you  measure  that? 

Martin  Armitage,  Director  of  Global  Infrastructure,  Unilever  Corp. 

A.  OUTLINE  OF  APPROACH 

To  date,  there  has  not  been  a  sueeessful  demonstration  of  the  methods  or  strategy 
necessary  to  successfully  implement  a  measurement  technique  that  can  indicate  the 
effectiveness  of  an  intranet.  By  soliciting  the  three  audiences  to  determine  which  metric 
area  is  of  greatest  impact  on  their  performance  and  satisfaction  (these  results  can  be 
ascertained  or  confirmed  through  analysis  of personasf,  metric  areas  from  each  segment 
can  then  be  prioritized,  logically  grouped  and  then  sub-grouped  with  specific  hard  and 
soft  metrics  supporting  each  group.  All  of  which  are  related  back  to  critical  business 
requirements  and  divided  among  the  audience  most  affected. 

1.  Matrix  for  Metrics  and  Prioritization 

Figure  7  below  shows  an  initial  baseline  estimation  of  the  top  several  metric 
areas,  which  segment  they  come  from  and  some  of  the  business  requirements  they 
sustain.  The  prioritization  of  these  metrics  is  based  on  the  metric  groups  most  impactful 
on  overall  value  from  a  business  management  perspective.  The  distribution  of  metrics  is 
of  additional  interest  from  a  theoretical  standpoint  because  at  least  one  metric  area 
originates  from  every  segment.  This  substantiates  the  assertion  that  more  metrics  need  to 
be  taken  from  more  than  the  Usage  segment  if  accurate  and  comprehensive 
measurements  of  effectiveness  are  to  be  indicated.  Nevertheless,  it  is  best  to  limit  the 
number  of  metrics  (the  baseline  focuses  on  seven)  to  include  only  those  that  directly 
correlate  to  a  business  benefit  or  else  the  analysis  may  become  overly  complicated, 
risking  confusion,  implementation  and,  therefore,  credibility. 

If  a  business  fails  to  understand  its  users,  then  it  will  probably  create  a  poor  product  or  service.  Personas 
(or  User  Archetypes)  are  a  way  of  depicting  users  of  products,  such  as  user  behavior  while  visiting  portal 
sites.  Using  personas  allows  development  teams  to  focus  more  on  design  and  apply  appropriate  effort. 
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H 

Access 
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Figure  7,  Ranking  of  Key  Intranet  Metrics  by  Segment  ^ 

2,  Methods  to  Substantiate  Measurements  and  Theoretical  Analysis 

There  are  three  advantages  to  specifying  down  to  single  metrics  logically  grouped 
(see  Table  2,  Example  of  One  Metric  Breakdown,  below  for  an  example  and  Appendix  D, 
Intranet  Portal  Metric  Breakdown,  for  a  far  more  complete  breakdown): 

•  When  the  group  consists  of  hard  metrics,  the  data  can  be  pulled  automatically 
and  routinely  over  time,  i.e.,  create,  query  and  retrieve  server  log  fdes  supporting  metrics. 


^  A  derived  metric  (denoted  in  the  table  by  “  *  ”)  can  become  a  hard  metric  within  the  Domain 
Infrastructure  segment  as  the  perceived  physical  and  software  interaction  between  key  elements  within 
the  constituent  of  DII  becomes  concretely  known  and  measured  with  precision,  examples  within 
Cross-Portal  Reference  and  Shared  Topography  above  are  the  ability  to  capture  all  unique  visitors  and 
information  maps  (akin  to  server  topography)  respectively. 
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•  Qualitative  data  can  be  derived  from  logically  grouped  quantifiable  data,  for 
example,  loyalty. 

•  When  metrics  are  closely  associated  to  user  behavior  and  business 
requirements,  analysis  of  the  results  lead  to  precision  and  powerful  modifications  which 
optimize  the  portal  more  in  line  with  what  needs  to  be  done  to  improve  its  effectiveness. 

Table  2,  Example  of  One  Metric  Breakdown,  on  the  next  page  presents  an 
example  of  the  thought  process,  reasoning  behind  it  and  how  to  interpret  the  results. 
Upper  portion  provides  the  what,  where,  who,  why  and  how: 

•  What  is  the  metric  grouping  that  has  been  prioritized  to  most  facilitate 
improvement  in  critical  business  requirements. 

•  Where  locates  at  the  level  or  segment  this  metric  is  most  appropriate,  for 
example  it  may  apply  to  the  entire  enterprise  or  to  mangers  of  a  portal  or  to  user. 

•  Who  refers  to  audience(s)  most  impacted  and  interested  in  these  results 

•  Why  outlines  the  business  process  and  is  further  delineated  into  three  sub¬ 
categories  of  Business  Issue,  Business  Question  and  Business  Significance: 

o  Business  Issue  refers  directly  to  the  critical  business  requirement  and 
the  supporting  derived  metrics. 

o  Business  Question  relates  the  main  concern  from  the  perspective  of 
either  or  combination  of  the  business  decision  maker,  portal  manager  or  technical  analyst. 

o  Business  Significance  is  the  explanation  to  Business  Questions. 

•  How  is  the  best  practice  of  applying  the  combination  of  metrics,  providing  a 
meaningful  name  for  it  and  then  calling  out  specifically  each  metric  in  that  group 

In  addition,  the  table  is  color  coded  to  match  the  model.  All  metric  areas  and  specific 
metrics  are  colored  to  represent  what  they  support  and  where  they  belong  in  the  lEEM  as 
they  are  used  with  that  metric  priority  or  What  category.  Eor  example,  the  color  pink  is 
used  for  all  metric  areas  and  metrics  that  relate  to  business  requirements  or  are  derived, 
yellow  for  Back-end  and  hard  metrics,  blue  for  Eront-end  or  soft  metrics  and  purple  for 
best  bets  or  practices. 
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WHAT:  UNIQUE  USERS  (Priority  No.  3) 

WHERE:  Enterprise 

WHO:  Portal  Owners  and  Managers 
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-  Top  Visitors  by  Hits  (Leads) 

-  Top  Visitors  by  Hits  Over  Time 
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-  Visits  by  Length  of  Visits 

*  All  of  the  above  for  Returning  Visitors 
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- 
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- 
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- 
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- 
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-  Top  Returning  Visitors  by  Hits  Over  Time 
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-  Top  Returning  Visitors  by  Leads  Completed 

-  Returning  Visitor  Visits  by  Length  of  Visits 

-  Returning  Visitor  Page  Views  Over  Time 

-  Top  Document  and  Content  Group  for 
Returning  Visitors  Over  Time 

Survey 

-  Internal  Returning  Visitor  Session  Activity 
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Where  do  my  visitors  come 

Identify  where  users  originate 
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Access 

overall  productivity  for  a 
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particular  region  organization, 
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-  Most  Active  by  Visits 
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-  Most  Returning  Visitors 
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-  Internal  Visitor  Session  Activity 
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Table  2,  Example  of  One  Metric  Breakdown  (taken  from  Appendix  D) 
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In  addition  to  being  grounded  in  a  theoretieal  framework,  this  approaeh  ean  be 
applied  praetieally  to  obtain  estimates  based  in  many  eommon  units  whieh  ean  be  traeed 
direetly  baek  to  speeifie  pages,  links,  design,  ete.  in  a  portal.  Thus,  how  to  go  about 
deriving  effeetiveness  of  portals  ean  be  operationalized  in  relatively  praetieal  ways. 
Moreover,  this  approaeh  is  not  reliant  on  any  partieular  software,  so  it  ean  be  applied  to 
any  network  regardless  of  its  network  operation  system  without  additional  eosts  to 
hardware  or  software,  exeept  saving  spaee  on  a  server  to  store  queries  of  log  files. 
Additional  eosts  will  be  ineurred  however  in  the  time  eharged  by  an  analyst  to  eonduet 
the  metries  or,  in  the  ease  of  a  larger  eorporation,  the  hiring  of  a  full  time  projeet  manager 
to  do  this  work  (the  extent  to  whieh  these  eosts  impaet  ROI  is  addressed  in  Chapter  V, 
Return  on  Intranet  Metrics  Investments  and  the  eomposition  and  manner  of  the  analyst(s) 
to  do  this  work  is  addressed  in  Chapter  VI,  Business  Intelligence  Team). 

B,  BRINGING  IT  TOGETHER 

However,  setting  a  seript  to  eull  data  periodieally  aeeording  to  a  baseline  of 
metries  is  not  suffieient  by  itself  Four  eritieal  aspeets  are  required  to  refine  this 
approaeh: 

1.  Analysis  of  User  Behavior 

Analysis  of  user  behavior  patterns  within  portals  on  different  sub-sites  and  pages 
is  both  informed  by  the  results  of  metries  used  and  foeused  surveys,  whieh  in  turn 
informs  and  refines  subsequent  metries  and  surveys  administered.  For  example,  after 
observing  that  there  is  a  high  number  of  visitors  abandoning  a  partieularly  important  site, 
user  behavior  must  be  taken  into  aeeount  via  a  foeused  survey  as  well  as  direet 
observation  and  analysis  of  other  metries  used  at  this  site  to  aseertain  the  reason  why  this 
is  oeeurring.  In  other  words,  until  sound,  lower-level  algorithms  are  written  to  explain 
this  behavior,  analytieal  skills  will  be  required  by  a  knowledgeable  “person  in  the  loop” 
analyst  (see  Seetion  VI.B.l:  Analyst). 
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Segmentation  provided  by  lEEM  along  with  the  attributes  that  are  linked  to  a 
user’s  loginID  add  to  the  value  of  all  intranet  portal  site  metries  beeause  it  enables  an 
organization  to  traek  disparate  behavior  in  different  segment  areas  by  different  audienees 
or  customer  groups.  Eor  example,  determining  how  first-time  users  perform  differently 
from  repeat  users  is  an  important  indication  of  how  successful  trial  efforts  will  be  at 
engendering  long-term  loyalty.  Conversion  rates  and  average  orders  will  be  lower  than 
repeat  customers,  but  narrower  gaps  will  be  easier  to  bridge.  Useful  comparisons  that  can 
be  made  which  take  into  account  the  combination  of  user  behavior  with  usage  statistics 
are: 

•  Percentage  of  traffic  and  page  views  from  new  vs.  repeat  visitors 

•  Average  browse  from  new  vs.  repeat  customers 

•  Conversion  rate  for  first-time  visitors  and  download  or  browse  time 

•  Conversion  rate  for  repeat  visitors  and  download  or  browse  time 

•  Page  views  for  new  vs.  repeat  customers  and  visitors 

The  end  result  of  analysis  may  dictate  page  re-design  (i.e.,  content  the  portal  owner  wants 
users  to  see  or  believes  they  need  is  not  being  found  because  it  is  a  few  layers  deep)  or 
new  content  added  (i.e.,  creating  new  links  or  placing  content  in  a  more  visible  location). 
Modifications  are  then  followed  by  the  same  set  of  metrics  to  see  if  there  is  an 
appreciable  change  in  patterns,  for  instance;  more  files  are  downloaded  or  users  are 
staying  at  the  site  for  longer  periods  of  time  than  before. 

The  metrics  must  remain  consistent  or  the  same  between  at  least  two  periodic 
measurements  to  accurately  monitor  the  impact  of  the  process  changes  made  as  a  result  of 
changes  to  sites.  The  fewer  the  changes,  the  more  likely  the  cause  and  effect  impact  can 
be  narrowly  isolated  and  tracked.  On  the  other  hand,  the  greater  the  number  of  changes, 
the  less  likely  all  impacts  can  be  accurately  ascertained  to  the  extent  tractable  value  can 
be  confidently  traced  to  specific  modifications.  The  application  of  metrics  is  a  balancing 
act  that  should  lend  itself  to  fewer  changes  needing  to  be  made  over  time  as  the  portal 
becomes  increasingly  optimized.  Eigure  8  below.  Balancing  changes  with  Cause  and 
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Effect,  illustrates  this  point,  showing  the  optimum  range  of  changes  that  would  normally 
occur  when  first  applied  to  a  newer  portal  and  then  over  time  as  the  same  portal  becomes 
more  optimized. 


Newer  portal 

Optimized  Portal 

Range  of  Changes 

Range  of  Changes 

Considered/Made 

‘'^3^^v,(Considered/Made 
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Figure  8,  Balancing  Changes  with  Cause  and  Effect 


Taken  a  step  further  after  modifications  are  made,  if  the  metrics  indicate  users  are 
staying  at  the  same  site  for  much  greater  lengths  of  time  and  in  greater  numbers,  either 
additional  surveys  or  metrics  may  be  called  for  to  determine  what  files  are  being  opened 
and  downloaded  to  better  understand  and  react  to  what  users  want.  This  may  be  further 
substantiated  with  clickstream  data  collection  in  some  instances  to  observe  behavior. 
Although  monitoring  the  every  move  of  an  employee  may  not  be  desirable,  it  is  certainly 
legal  and  implications  thereof  are  not  as  serious  when  it  is  administered  to  employees 
using  company  portal  resources.  Once  compiled,  the  results  from  these  metrics  could 
affect  caching  of  files  and  load  balance  of  servers  if  the  demand  or  user  behavior  is  great 
enough  to  indicate  the  server  is  not  configured  properly  for  the  type  of  load  it  is  receiving 
due  to  (metrics  related  analysis)  enhancements.  Alternatively,  it  could  simply  mean  that 
the  site  where  this  behavior  is  observed  in  large  numbers,  needs  to  be  updated  and/or 
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optimized  to  accommodate  demand.  All  in  all,  the  cycle  is  one  of  continued  refinement 
of  portals  via  metric  and  survey  analysis  to  meet  user  behavior,  expectations  and  needs. 

2.  Periodic  Soft  Metrics 

Periodic  soft  metrics  will  be  required  to  substantiate  and  confirm  what  groups  of 
harder  metrics  indicate.  From  time  to  time.  Front-end  metrics  need  to  be  applied,  i.e., 
usability  studies  and  surveys  from  the  Design  and  User  Experience  segments 
respectively.  As  suggested  in  the  analysis  of  user  behavior  above,  surveys  can  be  more 
focused  when  based  on  patterns  mapped  by  hard  metrics.  Surveys  based  off  analysis  of 
hard  metrics  are  more  succinct  and  provide  desired  feedback,  increasing  the  likelihood 
more  users  will  take  time  to  complete  them  and  with  less  frustration. 

a.  Surveys 

A  metrics  and  communication  program  is  critical  to  both  the 
communications  process  and  to  the  development  of  a  feedback  loop  so  that  IT  can  learn 
which  initiatives  provide  the  best  business  value.  If  the  portal  is  to  succeed  as  a  new 
paradigm  for  professional  computing,  it  must  be  able  to  recognize  and  adjust  to  ongoing 
changes  in  knowledge  workers’  information  needs  -  and  not  solely  with  usage  statistics. 
For  example,  the  portal  learning  loop  in  Figure  9  below  differs  from  other  architectural 
elements  in  that  it  is  not  concerned  with  a  specific  aspect  of  information  management,  but 
in  the  ongoing  effectiveness  of  the  portal  itself.  It  enables  the  portal  to  adjust 
heuristically  to  changes  in  the  organizational  work  and  information  environment. 

Surveys  provide  an  excellent  means  to  both  inform  and  be  informed  by  the 
learning  loop.  For  example,  raw  visitors’  metrics  might  indicate  that  an  infrequently 
visited  research  page  should  be  archived  or  discarded,  when  in  fact  a  single  recent  access 
may  have  been  the  critical  piece  in  securing  a  major  new  contract/revenue  stream  for  the 
organization.  Benefits  from  surveys  are  wide  spread  and  may  include  other  obvious  but 
overlooked  additions  such  as  providing  an  online  employee  manual  equipped  with  a 


50 


search  engine,  thereby  reducing  the  amount  of  time  people  spend  looking  for  the  manual 
and  information  within  the  manual  (see  Appendix  B,  Online  E-Survey  Example,  for  a 
general  semi-quantifiable  online  intranet  user  e-survey). 
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Figure  9,  Portal  Learning  and  Discovery  Loop 


Surveys  can  be  administered  dynamically  and  automatically  online  or  the 
old  fashioned  way  with  pen  and  paper.  The  advantages  of  conducting  surveys  online  are 
speed  and  convenience  for  all  parties  involved.  Provided  it  is  informed  and  succinct, 
knowledge  workers  will  learn  to  accept  them  if  they  perceive  it  helps  to  meet  their 
information  needs.  Surveys  that  warrant  user  clarification  are  far  more  practical  when 
based  in  conjunction  with  results  from  a  baseline  set  of  metrics  and  conversion  rations 
that  are  uniformly  applied.  Examples  include  interviewing  some  users  and  asking  them 
to  describe  the  products  (getting  their  descriptive  words)  and  then  observing  them 
complete  their  tasks  on  the  site.  Sometimes  a  survey  may  ask  knowledge  workers  to 
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name  the  eategories  in  whieh  they  expeet  to  find  the  produets.  At  other  times  a  survey 
will  look  at  the  pages  found  at  the  next  level  down  in  the  hierarehy  to  look  for  potential 
trigger  words.  These  words  ean  then  be  used  to  ereate  a  multi-level  eategory  lists  and 
assoeiate  them  to  expand  the  eurrent  taxonomy  for  broader  relevance  (within  portal 
context)  as  well  as  for  tagging  and  storing  new  corporate  information  and  data  formats. 

3,  Capture  and  Apply  Lessons  Learned 

Lessons  learned  need  to  be  captured  for  submission  as  business  rules  or  best 
practices  when  methods  are  proven  which  optimize  portal  performance.  Optimization 
examples  of  this  kind  include: 

•  Identifying  efficient  navigational  paths 

•  Pinpointing  frequently  visited  areas  of  the  site  as  potential  high  payoff  areas 

•  Planting  appropriate  keywords  within  key  pages  to  enhance  placement  within 
search  engines 

•  Identifying  which  referring  sites  are  most  effective 

Best  practices  that  result  in  new  policies,  regulations  and  business  rules  need  to  be 
supported  at  the  business  owner  level  and  higher  to  ensure  recommendations  are  enforced 
and  employed  enterprise-wide  with  a  smoother  finding  experience  for  knowledge 
workers  (see  section  VI.A,  Role  of  Business  Intelligence  Team). 

4.  A  Single  Reporting  Service 

A  single  reporting  service  needs  to  assist  portal  managers  in  administering  and 
interpreting  the  results  from  the  metrics  baseline.  This  could  be  done  by  the  creation  of  a 
product  manager  position  (the  person-in-the-loop)  responsible  for  driving  value  back  into 
the  corporate  intranet  by  maintaining  and  refining  the  metrics  baseline  through  analysis 
and  regular  interaction  with  both  users  and  portal  managers.  Without  this  entity,  the 
execution  of  metrics  for  the  intranet  will  most  likely  return  to  haphazard  and  freelance 
use  of  usage  metrics  focusing  on  peaks  of  volume  traffic,  much  like  it  always  has  been. 
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To  continue  measuring  intranet  performanee  through  sporadie  and  often  ill- 
eoneeived  groupings  of  metrics  —  taken  almost  exelusively  from  hard  metrics  in  the 
Usage  segment  —  will  do  little  to  drive  value  baek  into  intranets  or  provide  an  aecurate 
indication  of  the  performanee  and  effectiveness  of  portals  in  support  of  business 
requirements.  In  short,  it  would  be  near  failure  to  eontinue  the  status  quo  in  how 
intranets  are  measured  today.  A  single  repository  serviee,  based  on  a  praetieable  model, 
would  provide  an  improvement  to  eurrent  measurement  taking  teehniques  and  shed  light 
on  a  greater  awareness  of  the  importanee  and  role  of  all  segments  underlying  an  intranet. 
Some  of  the  benefits  that  should  be  aehievable  at  minimal  eost  due  to  this  eentralization 
and  tighter  eoordination  are; 

•  Domain  owners  experienee  inereased  traffie  by  target  audienees  and  higher 
levels  of  satisfaction  with  their  domain. 

•  Domain  owners  and  managers  are  more  effieient  with  their  own  resourees 
resulting  in  a  whole  that  is  more  effieient  than  the  sum  of  the  parts,  i.e.,  load  balanee, 
optimization  of  most  frequently  requested  pages,  downloads,  ete.  and  more  effeetive 
editorial  modifieations. 

•  Domain  owners  aetively  partieipate  in  the  development  and  adoption  of 
standards  and  best-praetiees,  and  are  rewarded  and  reeognized  for  their  work. 

•  Enterprise  experiences  inerease  effieieneies  (eost  savings)  due  to  elimination 
of  redundaneies  and  inereased  use  of  shared  serviees,  e.g.  people  databases  and 
preseriptive  arehiteetural  guidanee. 

•  Inereased  employee  satisfaetion  with  seareh  and  navigation. 

One  other  signifieant  impaet  to  a  eorporation  of  a  single  eolleetion  and  reporting 
souree  is  the  benefit  of  being  able  to  direetly  develop  and  refine  its  own  model  and 
database  from  immediate  aeeess  to  data.  The  hypothesis  is  that  predietors  (the  metrics 
used)  get  better  with  praetiee  and  loeal  data  provides  the  best  predictive  base  because  it  is 
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easier  to  filter  out  extraneous  variables  and  to  foeus  on  key  variables.  The  issues  and 
benefits  of  a  single  reporting  serviee  are  addressed  in  greater  detail  in  Chapter  VI, 
Business  Intelligence  Team. 

C.  PERIODIC  REVIEW 

Hard  and  soft  metries  taken  together  with  eonsideration  given  to  their  strengths 
and  weaknesses  allows  an  enterprise  to  make  informed  deeision  on  the  investment  in,  or 
the  ongoing  value  of  its  data  warehouse  and  portal  system.  Aehieving  sueeess  through 
the  use  of  any  performanee  metrie  will  depend  as  mueh  on  how  well  it  is  applied  as  it 
does  on  when  it  is  used.  Studies  based  on  samples  and  averages  over  time  ean  make  for 
easier  and  more  eredible  eomparisons.  Henee,  eontinuous  benehmarking  should  be 
instantiated  to  eonfirm  and  eorreet  baseline  measurements  and  eonversion  rations  through 
periodie  (i.e.,  monthly,  quarterly,  annually)  portal  status  reviews  that  measure  progress 
against  previous  baseline  results.  For  these  reviews,  portal  owners  should  use  the  metrics 
to  determine  which  roles  and  content  are  being  underserved  by  the  portal  and  which 
processes  could  correct  this  and  better  leverage  the  portals  capabilities. 

Three  different  scenarios  would  likely  play  out  when  an  accepted  baseline  of 
metrics  is  applied  to  an  enterprise:  introductory,  intermediate  and  developed 
implementation.  The  parties  involved  (see  Chapter  6,  Business  Intelligence  Team) 
determine  all  relevant  metrics  possible  within  the  areas  that  make  up  effectiveness  and 
then  prioritize  them  in  importance  with  regard  to  relevance,  balance,  accuracy  and 
audience.  For  example,  this  work  is  based  in  part  from  data  collected  from  Site  Audit 
and  Technical  User  Information  Surveys  applied  against  a  baseline  set  of  metrics  and 
conversion  ratios,  such  as  those  of  lEEM  (see  Appendix  D,  Intranet  Portal  Metric 
Breakdown). 

In  the  introductory  scenario  pre  and  post-pilot  tests  to  measure  underlying  areas  of 
effectiveness  using  the  prioritized,  combination  of  metrics  based  on  lessons  learned  and 
results  from  surveys  are  scheduled  to  be  conducted  to  find  the  right  fit  and  rhythm  in 
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metrics  selected,  i.e.,  how  and  when  they  can  be  most  conveniently  and  operationally 
applied.  Afterwards,  there  needs  to  be  a  confirmation  meeting  with  Portal  Manager  and 
at  least  one  champion  BDM  to  establish  agreement  with  the  measurement  approach  and 
analysis  to  be  conducted  as  well  as  to  coordinate  resources  in  order  to  conduct 
measurements  and  collect/compile  data..  After  the  first  pilot  is  administered  to  a  selected 
group  of  portals  (it  may  not  be  wise  to  apply  the  pilot  to  the  entire  enterprise  until 
unforeseen  kinks  are  worked  out),  data  is  collected  as  much  by  automation  as  possible 
and  then  compiled  and  collated  for  analysis.  Based  on  the  analysis,  a  few  tractable 
changes  to  processes  within  the  portals  are  made  for  further  observation  with  the  intent  to 
improve  productivity. 

Scenario  One  -  Introductory 
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Figure  10,  Implementation  Scenarios  for  Applying  a  Baseline  Metrics 


The  second  post-pilot  collection  of  data  then  takes  place  over  the  same  time 
period,  usually  30  days,  with  the  same  set  of  metrics  and  conditions  as  before,  with 
exception  of  the  process  modification(s)  made.  After  these  results  are  processed,  the 
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analysis  may  take  longer  to  determine  the  impact  of  the  changes  and  the  extent  to  which 
they  improved  the  odds  to  increase  value.  In  some  cases,  this  analysis  may  need  to  be 
facilitated  by  vendor  support  until  the  company  is  able  and  willing  to  do  this  nature  of 
work  by  itself.  Following  the  analysis  period,  the  results  need  to  be  present  by  the 
analysis  or  business  intelligence  team  to  all  the  portal  managers  and  owners  of  the  portals 
monitored  as  well  as  their  adjunct  BDMs  if  possible  to  provide  executive  level  reaction 
and  feedback.  At  the  end  of  this  meeting,  there  should  be  a  wider  and  better 
understanding  of  the  necessity  and  impact  of  a  baseline  set  of  metrics  and  corresponding 
conversion  ratios  grounded  in  the  lEEM. 

Subsequent  measurements  need  to  be  taken  periodically  and  preparations  for 
improved  and  expanded  runs  of  the  metrics  program  may  eventually  become  a  regular 
job  for  selected  employees  at  different  levels.  Over  time,  less  frequent  measurements  are 
needed  as  the  implementation  reaches  the  developed  scenario,  whereby  measurements  are 
taken  semi-annually  (see  Eigure  10  above.  Implementation  Scenarios  for  Applying  a 
Baseline  Set  of  Metrics).  Providing  managers  at  multiple  levels  with  a  practical,  valid 
and  reliable  way  to  monitor  portal  activity  and  affects  on  its  processes  is  only  part  of  the 
battle  that  needs  to  be  waged  by  the  business  intelligence  team.  Elltimately  such 
methodologies  must  pass  muster  with  the  accounting  and  financial  communities  and 
those  agencies  that  regulate  them  (Baskerville,  1999,  pp.  46-47).  An  lEEM  baseline 
metric  approach  would  allow  collaboration  among  all  concerned  parties  such  that 
interventions  based  on  the  use  of  this  methodology  could  be  tracked  over  time  as  the 
parties  attempt  to  reach  consensus  on  the  meaning  of  the  outputs  of  such  analysis.  It  is 
only  then  that  such  metrics  will  have  a  lasting  impact.  This  is  precisely  why  a  business 
intelligence  team  needs  to  be  formed  and  is  discussed  at  greater  length  in  Chapter  VI. 


D,  SUMMARY 

The  intranet  is  the  most  measurable  medium  ever.  With  respect  to  legal  issues 
and  the  compliances  more  easily  demanded  within  the  internal  operations  for  any 
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organization,  it  is  more  measurable  than  the  Internet.  Yet  organizations  of  all  sizes  and 
types  fail  to  measure  its  full  impact  because  it  is  considered  either  too  hard  or  not  a 
priority.  To  date,  there  has  not  been  a  successful  demonstration  of  the  methods  or 
strategy  necessary  to  successfully  implement  a  measurement  technique  that  can  indicate 
the  effectiveness  of  an  intranet.  By  soliciting  the  three  audiences  to  determine  which 
metric  area  is  of  greatest  impact  on  their  performance  and  satisfaction,  metric  areas  from 
each  segment  can  then  be  prioritized,  logically  grouped  and  then  sub-grouped  with 
specific  hard  and  soft  metrics  supporting  each  group.  All  of  which  are  related  back  to 
critical  business  requirements  and  divided  among  the  audience  most  affected. 

As  the  intellectual  capital  builds  in  corporate  portals,  more  investments  will  be 
made  to  enhance  them  and  make  them  greater  enablers.  Executives  will  demand  from 
their  IT  that  they  implement  a  program  of  metrics  where  each  major  initiative  has  defined 
goals  and  metrics  to  indicate  whether  or  not  these  goals  have  been  obtained.  To  translate 
these  metrics  into  financial  terms  using  standard  conversion  factors,  such  as  the  cost  to 
the  company  of  each  employee  saved,  the  value  of  time  saved,  increased  revenue  per 
customer  or  transaction,  or  the  savings  in  time  and  money  from  fewer  defects  will  require 
far  greater  appreciation  of  intranets  and  the  metrics  needed  to  measure  them  than  is 
exercised  in  public  and  private  sectors  today.  Moreover,  IT  will  be  required  to 
communicate  these  financially  oriented  results  on  a  regular  basis,  something  many  IT 
professionals  today  are  not  versed  at  doing.  The  next  two  chapters  on  ROI  and  business 
intelligence  teams  focus  on  how  to  do  this  and  by  whom,  respectively. 
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V.  RETURN  ON  INTRANET  METRICS  INVESTMENT 


Return  on  investment  should  not  be  the  key  benchmark  in  deciding  on  a 

company's  IT  spending. 

Douglas  Busch,  Chief  Information  officer,  Intel 

A,  BACKGROUND 

ROI  has  become  a  big  buzzword  in  IT,  yet  few  companies  are  tracking  ROI  in  a 
consistent  and  standard  manner.  Enterprises  are  increasingly  scrutinizing  their  IT 
spending  and  proponents  of  e-business  projects  must  go  to  much  greater  lengths  to  justify 
any  spending  than  they  were  required  to  do  during  the  dotcom  boom.  As  a  standard 
fallback,  companies  rely  on  traditional  return  on  investment  (ROI)  metrics  to  make  e- 
Business  funding  decisions.  Evaluating  the  potential  return  on  an  IT  investment  can  be 
fairly  straightforward— at  least  in  theory.  In  financial  terms,  ROI  means  profit  divided  by 
investment,  expressed  as  a  percentage  (King,  2002,  p.41).  But  within  that  definition, 
there  is  a  lot  of  room  for  interpretation  and  pitfalls: 

•  Does  the  initiative  fit  strategically? 

•  Does  the  initiative  support  functional  objectives? 

•  Does  the  initiative  incorporate  opportunities  for  process  improvement? 

•  Does  the  underlying  technology  fit  the  infrastructure? 

•  Are  ROI  numbers  based  on  reasonable  assumptions? 

By  focusing  solely  on  what  is  quantifiable  in  terms  of  dollars  and  cents  to  answer  these 
types  of  questions,  companies  risk  being  precisely  wrong  instead  of  being  approximately 
right.  The  problem  being  that  it  is  difficult  to  translate  many  benefits  into  hard  cash 
equivalents.  This  applies  equally  to  IT  investment  in  a  business  context.  The  tendency  is 
to  apply  a  strictly  quantitative  ROI  when  in  fact  true  ROI  is  both  quantitative  and 
qualitative.  Nevertheless,  corporate  managers,  who  subscribe  to  the  mantra,  "If  you  can't 
measure  it,  you  can't  manage  it",  want  quantifiable  statistics  to  more  concretely 
demonstrate  that  their  departmental  effort  is  pulling  and  financing  its  share  of  the  weight 
in  the  corporation. 
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Return  on  investment  is  popular  because  it  is  a  simple  concept  that  everyone  can 
understand.  ROI  and  its  cousins,  Net  Present  Value,  Internal  Rate  of  Return,  Payback 
Period,  and  Economic  Value  Added  are  concepts  that  executives  have  traditionally  used 
to  measure  performance  (see  Appendix  E,  Common  Approaches  to  IT  ROI,  for 
advantages  and  disadvantages  to  these  and  other  techniques  de your  in  calculating  ROI  for 
IT).  These  metrics  are  certainly  useful  but  they  fall  short  of  providing  a  complete 
financial  picture  for  business  planning.  With  Intranet  ROI  seen  as  a  metric  that  ranks  a 
technology  investment  in  relation  to  other  company  investments,  attempts  are  made  to 
evaluate  intranet  expenses  in  terms  of  cost  savings  that  are  attributable  to  investments  in 
business  process  automation.  ROI  therefore  would  seem  like  a  logical  way  to  assess 
intranet  related  payoffs.  In  practice  however,  traditional  ROI  metrics  fail  to  measure  the 
value  created  by  intranets  —  forcing  business  managers  at  multiple  levels  to  make  e- 
Business  funding  decisions  based  on  gut  feelings,  rather  than  tracking  ROI  in  a  consistent 
and  standard  manner  with  the  aid  of  Web  metrics  within  a  business  value  framework. 

Despite  having  high  expectations  for  portals,  only  a  minority  of  firms  report 
having  formal  metrics  for  documenting  portal  benefits  -  and  virtually  all  of  these  are  for 
Internet  and  not  intranet  portals.  Astoundingly,  51%  percent  of  firms  don’t  have  any 
metrics  to  prove  portal  benefits  and  another  20%  don’t  know  if  they  have  any  ROI  related 
metrics  at  all  for  portals  (Gillet,  2001,  p.  6).  In  addition,  approximately  66%  of  IT 
managers  believe  ROI  is  an  appropriate  metric  only  “sometimes”  for  an  IT  site  (Upton, 
2001),  see  Eigure  9,  ROI  Measured  on  IT.  Moreover,  these  opinions  on  the  usefulness  of 
ROI  for  IT  are  based  largely  on  Internet  e-Commerce  sites,  not  in  relation  to  intranets 
which  are  perceived  more  as  “sunk  costs”  of  doing  business.  All  the  same,  the  difficulty 
of  determining  valid  ROI  for  Internet  e-Commerce  sites  is  another  reason  why  IT 
professionals  and  BDMs  often  avoid  applying  ROI  to  their  intranet  portals. 
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ROI  Measured  on  eCommerce  Sites  Is  ROI  an  appropriate  Metric  for  IT  Sites? 

Don't 

Know  Vos 


Figure  II,  ROI  Measured  on  IT 


I,  Intranet  Measurement  Misconceptions 

If  enterprises  are  to  better  realize  more  productivity  from  their  portals,  they  need 
to  understand  why  it  is  a  fallacy  to  believe  measuring  portal  value  is  either  undesirable  or 
undoable.  There  are  several  reasons  that  perpetuate  this  false  perception; 

•  Cost  cutting  is  so  much  easier  to  understand  and  measure  than  effectiveness 
that  it  almost  always  tends  to  gain  prominence,  despite  a  firm's  original  best  intentions. 

•  A  belief  that  the  accuracy  of  data  in  many  ROI  models  is  so  limited  there  is  no 
point  in  calculating  them.  For  example,  if  one  is  making  up  numbers  to  begin  with,  it's 
not  really  going  to  help  decide  whether  an  IT  project  makes  sense. 

•  Many  applications  are  so  inexpensive  to  develop  and  deploy  that  companies 
often  assume  they'll  get  a  return  on  their  investments  or  they  justify  these  relatively  small 
investments  by  pointing  to  intangibles,  such  as  improved  employee  morale  from  having 
easy  access  to  their  human  resources  and  401(k)  records  and  better  workforce 
collaboration,  resulting  in  quicker  time  to  market. 

•  Managers  of  intranet  portals  and  their  bosses  generally  believe  they  don't  need 
anyone  to  convince  them  of  its  business  value.  Managers  and  their  knowledge  workers 
know  that  the  communities  build  loyalty,  give  valuable  feedback,  and  contribute  to 
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increased  sales.  Thus,  IT  projeet  teams  shouldn’t  jump  through  finaneial  hoops  trying  to 
eost  justify  essential  investments  that  are  “no-brainers.” 

•  Depending  on  the  size  of  the  company,  it  is  not  worth  the  eost  or  time  to 
determine  the  ROI  for  an  intranet  upgrade  or  enhancement  (effectively  condueting  an 
ROI  on  the  ROI).  Though  ROI  ealeulations  are  different  for  different  industries  and 
eompanies,  as  a  general  rule  it  is  not  always  eost  effeetive  for  small  eompanies  to  conduet 
ROI  on  modest  modifieations  to  their  intranet  portals.  Many  IT  organizations  go  to  the 
trouble  of  doing  the  ROI  math  only  for  expensive  projeets. 

Enterprises  laek  a  business  strategie  model,  an  affiliated  metries  baseline  and  a  Business 
Intelligenee  Team  (BIT)  to  eonduet  the  neeessary  metrics  analysis  to  better  discern  and 
balanee  appropriate  quantitative  and  qualitative  metrics.  As  a  backdrop  to  the  above,  IT 
initiatives  are  inereasingly  beeoming  so  important  that  eompanies  are  either  not 
evaluating  ROI  or  they  are  realizing  the  need  to  develop  new  ways  to  measure  ROI  to 
take  into  aeeount  a  projeefs  strategie  value,  i.e.,  enhaneements  to  their  intranet  portals 
and  the  metries  needed  to  prove  and  disprove  improvement. 

In  essence,  strategie  IT  investments  that  improve  the  overall  effieieney  of  the 
enterprise  will  make  the  business  more  produetive,  saving  money  and  improving  the 
responsiveness  of  the  whole  eompany.  Thus,  improving  the  performanee  of  an 
enterprise’s  intranet  portals  is  of  great  signifieanee  beeause  the  greatest  impact  of  portals 
is  improved  produetivity.  Consequently,  eompanies  are  going  to  have  to  start  believing 
in  —  and  doing  —  this  hard  work  of  determining  productivity  benefits  because 
produetivity  is  the  most  important  pieee  of  ROI  analysis  (Koeh,  2002,  p.  3).^  Because  it 
is  hard  to  quantify  all  intranet-related  investment  benefits  and  savings  to  the  business, 
these  important  faetors  get  short  shrift,  even  though  they  ean  result  in  big  savings  that  ean 
have  a  direet  impaet  on  the  profitability  and  overall  effeetiveness  of  a  business.  Though 
traditional  aeeounting  methods  make  it  awkward  to  absolutely  quantify  IT's  value,  this 

^  In  a  Darwin  survey  of  75  CEOs  from  a  broad  range  of  companies,  increased  productivity  was  cited  as  by 
far  the  most  important  factor  (87  percent,  far  ahead  of  the  next  highest  factor,  customer  satisfaction)  they 
use  to  determine  if  IT  is  delivering  full  value  for  the  money  spent. 
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doesn't  mean  the  issue  of  an  IT's  value  should  be  ignored.  Business  owners  and  finance 
executives  alike  need  to  be  mindful  that  ROI  methodologies  tend  to  focus  on  short  term 
quantifiable  justifications,  while  ignoring  the  strategic  role  IT  plays  driving  new 
opportunities  for  the  business. 

2,  Qualitative  Inputs  and  Raw  Benefits 

The  nature  of  these  metrics-related  challenges  with  respect  to  value  should  force 
business  owners  to  find  credible  ways  to  rely  more  on  qualitative  inputs.  As  the  role  of 
IT  becomes  increasingly  strategic  to  the  success  of  a  corporation,  a  new  set  of  metrics 
needs  to  be  applied  to  investments  that  attempt  to  measure  IT's  ability  to  enable  increased 
revenue  or  faster  growth  for  the  business.  For  instance,  if  an  investment  improves  the 
time  to  market  for  a  critical  new  product  or  service,  it  could  be  said  to  have  added 
revenue  to  the  firm.  In  many  situations  getting  to  market  early  results  in  a  big  advantage 
both  in  terms  of  more  overall  revenue  in  the  life  of  the  product  or  service  and  in  a  period 
of  higher  margins  before  competitors  create  downward  pressure  on  prices.  In  effect, 
portals  help  meet  new  challenges  to  compete  on  the  basis  of  time  —  not  for  the  sake  of 
speed  for  its  own  sake,  but  because  profitability  in  markets  is  increasingly  available  only 
to  early  entrants  who  can  forge  brand  and  business  dominance  (a  parallel  could  also  be 
made  for  a  government  or  military  in  reducing  their  decision  time  cycles). 

Return  on  intranet  metric  investment  (ROIMI)  has  more  credibility  and  is  clearer 
when  it's  stated  in  raw  benefits,  which  are  sometimes  non-quantifiable,  rather  than 
translated  into  short-term  return  dollars.  The  numbers  tell  the  story,  but  not  the  whole 
story:  Some  benefits  may  not  be  quantifiable  today  in  terms  of  dollars,  for  example  ease 
of  use,  competitive  advantage,  customer  loyalty,  etc.  These  benefits  are  worth  including 
in  the  value  story,  despite  the  risk  of  companies  still  ending  up  with  ROI  results  that  are 
not  100%  quantifiable.  The  challenge  is  to  prevent  the  translation  from  becoming  fuzzy 
and  losing  some  audiences. 
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Calculating  ROI  requires  a  eonsiderable  amount  of  data,  eonsistent  standards, 
baselines  within  a  eompany,  and  at  least  some  financial  expertise.  Even  with  sueh  input, 
the  end  results  are  approximate  and  ean  be  manipulated.  Moreover,  intranet  ROIs  from 
different  eompanies  are  unique  and  returns  may  differ  beeause  they  refleet  rapid  changes 
in  teehnology  and  knowledge  worker  behavior  whieh  varies  from  eompany  to  eompany 
as  do  their  expertise  levels  and  fields.  Even  with  these  restrietions,  the  proeedure  of 
pursuing  ROI,  in  whatever  format,  forees  a  eompany  to  think  about  the  best  ways  to 
measure  sueeess  -  speeifieally,  the  hard,  soft  and  derived  metries  that  matter  most  in 
measuring  the  extent  to  whieh  business  values  are  supported  and  aehieved. 

B,  AXIOMS  AND  DRAWBACKS 

Despite  best  intensions,  ROI  ean  be  misleading  and  its  limitations  need  to  be 
reeognized  and  addressed.  Reeognizing  these,  a  deeper  analysis  of  the  IT  portfolio  ean 
find  truly  signifieant  eontributions  to  the  eorporate  health  and  well-being  that  should  be 
ealeulated  and  eommunieated.  In  the  past,  IT  organizations  used  to  be  viewed  as  a 
neeessary  expense  for  a  business.  Today,  however,  they  are  often  viewed  as  an  important 
strategie  asset  to  a  eompany's  future  sueeess.  While  many  businesses  are  foeused  on  eost 
eontrol,  IT  groups  are  often  foeusing  on  how  to  generate  growth  for  their  business  by 
funding  projects  to  help  their  eompany  reaeh  new  eustomers  and  work  more  effieiently 
with  its  existing  partners.  Perhaps  not  so  surprisingly,  only  about  8.33  pereent  of  IT 
spending  is  perceived  in  some  eireles  to  provide  ineremental  benefit  to  the  organization 
(Gliedman,  2000,  p.  2). 

The  reality  is  the  benefits  often  far  exeeed  this  pereeption;  otherwise  eontinued 
large  investments  into  it  would  not  be  made.  Eike  metries  for  intranets,  intranet  ROI  ean 
also  be  eounter  intuitive.  The  following  table  outlines  seven  shorteomings  that 
eompanies  tend  not  to  take  into  aeeount  when  ealculating  ROI  on  IT  and  its  impaet: 
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Axiom 

Drawback 

Impact 

1.  ROI  must  capture  all  costs, 
direct  and  indirect,  associated 
with  the  project/technology, 
including  products  and 

services  devoted  to  direct 
support. 

•One  size  doesn't  fit  all. 

•Applying  ROI  to  every  activity  won't  work. 

•The  human  factor  of  computer-assisted  work  is 
neglected  by  the  ROI  model. 

•One  metric  can’t  characterize  the  entire  IT  value. 

ROI  is  useless  when  it  comes  to  evaluating  risk, 
flexibility,  and  intangible  benefits-factors  that  have  a 
critical  impact  on  business  and  should  be  factored  into 
spending  decisions.  In  addition,  ROI  can’t  calculate 
valuable,  intangible  qualities. 

2.  ROI  must  be  based  on 
quantifiable  results.  The 
baseline  metric  is  the  dollar, 
often  measured  through 
calculating  time  saved 
(time  is  money). 

•ROI  is  both  quantitative  and  qualitative. 

Although  ROI  numbers  may  appear  concrete,  they 
can  be  misleading.  Unless  ROI  analysis  is  applied 
honestly,  there  is  ample  room  for  delusion.  The 
greatest  danger  is  the  “concrete”  and  “measurable”' 
driving  the  significant  out  of  the  analysis.  Because 
intranet  applications  are  internally  focused,  it's 
difficult  to  get  an  ROI  and  is,  therefore,  neglected. 

3.  ROI  must  be  based  on 
observable  results. 

Automation  has  business 
value  only  when  we  can  see 
the  results  such  as  faster 
production  or  higher  quality. 

•E-business  projects  often  follow  the  law  of 
unintended  consequences  because  they  cross 
functional  and  enterprise  boundaries,  and  may 
produce  payoffs  in  ways  that  were  hard  to  imagine 
at  the  outset. 

Taking  an  accountant's  view  of  IT  priorities  could 
actually  be  counterproductive,  because  a  spreadsheet 
doesn't  tell  the  whole  story.  In  fact,  some  of  the  IT 
projects  that  impact  business  the  most  can't  be 
measured  easily,  if  at  all,  some  experts  now  say. 
However,  the  only  way  to  grow  the  bottom  line  on  a 
sustainable  basis  is  to  grow  the  top  line,  which  is  easy 
to  ignore  if  every  project  is  measured  on  tangible 
ROI. 

4.  ROI  emphasizes  tangible 
payoffs  that  can  be  measured 
in  financial  terms.  Often,  the 
easiest  to  measure  returns  are 
bottom-line  improvements 

rising  out  of  cost  reductions. 

•ROI  is  a  metric  that  favors  cost-saving  projects. 

ROI  tends  to  favor  projects  that  result  in  cost 
avoidance,  at  the  expense  of  projects  that  promise 
revenue  growth.  This  is  particularly  acute  in  ROI  for 
intranet  initiatives  because  they  are  likely  to  miss  at 
the  outset  subsequent,  positive  impacts. 

5.  Investments  in  k-worker 
and  partner-facing  initiatives 
result  in  more  effective 
collaboration  and  translate 
into  important  productivity 
benefits  for  all  sides. 

•ROI  measures  only  the  returns  that  the  company 
sees  within  its  internal  operations. 

By  ignoring  the  value  created  for  partners  and 
customers,  ROI  may  be  missing  the  real  point  of 
e-Business  (and  the  very  idea  behind  creating  a 
corporate  portal  in  the  first  place).  For  example,  as  a 
parallel  approximately  two-thirds  of  the  overall 
benefit  of  a  retail  website  cannot  be  accounted  for  by 
online  transactions  alone. 

6.  ROI  calculations  for  cost¬ 
saving  projects  are  more 
accurate  because  the 

enterprise  already  has  the  data 
needed  for  the  equation. 

•When  calculating  ROI  for  a  revenue-generating 
project  like  intranet  portals,  estimates  are  often  used, 
which  makes  the  ROI  calculation  less  accurate. 

The  result  is  that  revenue-generating  projects  are  at  a 
disadvantage  if  they  are  competing  against  cost- 
savings  projects  based  on  ROI.  Furthermore,  not  all 
data/outputs  can  be  known  ahead  of  time  with  regard 
to  creativity  and  the  volatility  of  content  in  data- 
warehouses  and  the  impact  of  knowledge  applied  to  it. 

7.  Most  e-business  initiatives 
take  time  to  get  accepted  and 
widely  adopted.  Declaring 
failure  or  success  is  based  on 
a  three  or  a  six-month  time 
period  (time  is  money). 

•There  can  be  significant  time  lags  between  the 
benefits  that  will  impact  revenue  and  the  revenue 
metrics  themselves.  The  lag  time  may  be  six  months 
or  more  for  larger  projects.  Therefore,  companies 
examining  only  the  metrics  most  closely  tied  to 
revenue  risk  cutting  off  projects  for  which  positive 
return  may  be  just  around  the  comer. 

•Not  all  returns  are  financial  returns  in  the  short  mn, 
although  they  eventually  may  impact  financially. 

Most  e-business  projects  result  in  payoffs  on  multiple 
dimensions.  It's  a  tactical  approach  to  ROI.  The 
shorter  the  study,  the  shorter  the  ROI  basis,  the  more 
isolated  it's  going  to  be  and  the  less  meaningful  it's 
going  to  be  in  the  overall  strategy.  It's  significant  but 
it  needs  to  be  put  into  the  larger  context. 

Table  3.  Axioms  and  Drawbacks 
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In  effect,  in  the  rapidly  changing  world  of  IT,  ROI  is  ROI  —  except  when  it’s  not. 
While  CIOs  say  the  payback  on  most  IT  projects  can  be  measured  in  dollars,  many 
utilitarian  but  necessary  efforts,  like  infrastructure  upgrades  and  installing  and  supporting 
collaborative  applications,  don’t  translate  easily  and  those  projects  are  not  given  full 
credit  because  of  the  inability  (and  in  some  cases,  the  non-necessity)  to  attribute  any 
intellectual  gains  to  new  technology.  For  example,  in  the  real  world  financial  ROI 
calculations  may  be  0%,  but  the  overall  return  of  all  the  measures  can  easily  be  well  over 
100%.  Consequently,  the  ROI  model  needs  to  be  extended  to  be  more  comprehensive 
and  dynamic  to  take  into  account  time  to  value  while  factoring  traditional  return  on  asset 
analysis. 

Strictly  quantitative  ROI  logic  for  intranet  portals  and  IT  is  not  sufficient.  For 
example,  if  every  chief  financial  officer  discounted  productivity  gains,  companies  would 
not  have  PCs  on  their  knowledge  workers’  desks  because  they  don't  have  positive  ROI 
without  the  productivity  gains  and  are  too  costly  to  manage  relative  to  the  hard  savings 
they  provide.  Thus,  executives  should  exercise  caution  when  demanding  “balance  sheet 
efficiency”  on  ROI  calculations  for  IT  investments. 

Though  there  are  many  ways  to  express  standard  measures  of  return,  when 
traditional  ROI  conventions  don't  tell  the  whole  story,  business  managers  and  vendor 
alike  will  often  place  greater  emphasis  on  soft  benefits  —  like  employee  satisfaction, 
improved  visibility,  improved  knowledge  transfer,  and  dozens  of  other  assets  that  cannot 
be  measured  in  hard  numbers  —  plausibly  sure,  but  with  no  attempt  to  put  a  dollar  figure 
on  the  “smiley  faces”.  Thus,  to  overcome  this  weakness  when  calculating  ROI  for  IT 
projects,  decision  makers  should  consider  all  techniques  available  to  credibly  and  better 
measure  the  overall  impact  of  the  investment;  they  must  look  beyond  ROI. 

C.  BEYOND  ROI 

Even  in  some  of  its  strictest  applications,  ROI  is  far  from  being  a  perfect 
calculation.  It  is  understandable  then  that  an  enterprise  would  be  tempted  to  judge  the 
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success  of  its  intranet  portal  initiatives  on  its  traffie  volume  statisties  alone;  they  are  the 
only  value  related  indieators  that  ean  be  exaetly  quantified  that  a  portal  brings  to  a 
business,  i.e.,  showing  how  many  knowledge  workers  visit  the  site  over  time  and 
therefore  providing  some  level  of  value.  Beeause  an  intranet  does  have  a  measurable 
impaet  on  traditional  businesses,  eompanies  must  embraee  a  set  of  metries  that  gives  the 
portal  initiatives  eredit  not  only  for  its  online  knowledge  workers,  but  also  its  overall 
eontribution  to  the  eorporation  at  large  in  improving  its  eompetitive  advantage. 

The  key  to  eseaping  the  ROI  trap  is  to  think  strategieally  about  the  outeomes  and 
the  payoffs  from  intranet  portals  by  foeusing  more  broadly  on  business  value  and  meeting 
the  needs  of  knowledge  workers.  The  anehor  for  any  e-business  projeet  should  be  the 
value  ereated.  Foeusing  on  value  ereated  for  eustomers  as  opposed  to  eost  savings  for  the 
eompany  by  supplementing  speeulative  fmaneial  outeomes  (some  of  whieh  are  dubiously 
eontrived)  and  quantitative  metries  with  qualitative  ones  that  are  rationally  more  strategie 
in  nature  (and  eolleetively  provide  more  important  leading  indieators  to  gauge  the 
eompetitiveness  of  the  business)  is  fundamental  in  looking  beyond  ROI.  Measures  sueh 
as  customer  and  partner  satisfaetion,  eustomer  loyalty,  response  time  to  eompetitive 
aetions  and  improved  responsiveness  are  examples  of  these  soft  measures.  Subjeetivity 
in  these  “intangible”  measures  ean  aetually  be  quite  objeetive  if  used  eonsistently  over 
time.  For  example,  eustomer  satisfaetion  measured  eonsistently  on  a  five-point  seale 
survey  can  be  an  objeetive  basis  for  measuring  the  performanee  of  eustomer- faeing 
initiatives  (Sawhney,  2002,  p.  37),  see  Appendix  B,  Online  e-Survey  Example. 

A  variety  of  ROI  teehniques  exist  for  managers  to  assess  the  value  of  intangible 
benefits.  "Business  value  added"  and  "intangible  value"  are  both  eoneepts  used  to 
deseribe  how  IT  dollars  support  key  business  goals  that  aren't  easily  quantified. 
Similarly,  "return  on  opportunity"  helps  eompanies  examine  top-line  growth  potential 
rather  than  foeusing  on  eost  savings.  "Return  on  relationship"  aeknowledges  the 
intangible  nature  of  an  e-business  by  measuring  whether  relationships  produee  direet  or 
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indirect  returns  to  a  company,  such  as  speed-to-market.  By  contrast,  strict  financial  ROI 
approaches,  though  straightforward,  can  easily  stifle  opportunities  to  create  competitive 
advantage  and  ignore  the  impact  of  intangible  benefits  altogether. 

Best-practice  organizations  realize  that  outcomes  are  more  important  than  outputs. 
Tracking  the  interplay  between  pain  points  in  processes  and  the  subsequent  impact  of 
modifications  to  intranet  portals  to  affect  business  in  them  based  on  metrics  that  tie  back 
to  key  business  drivers  is  the  most  critical  yet  underused  measure  to  improve 
performance  outcomes.  Metrics  must  tie  back  to  the  original  business  drivers  so  more 
credible  and  comprehensive  justifications  can  be  provided  when  the  intranet  project  is 
proposed.  Consequently,  executive-level  ROIs  should  emphasize  outcomes  rather  than 
hit  rates,  which  is  the  norm  today.  New  intranet  applications  often  provide  new  ways  of 
working,  and  companies  should  approach  ROI  in  this  area  with  sensible  notions  of  “Does 
the  opportunity  justify  the  investment”  or  “Is  perceived  value  greater  than  the  cost?” 
Alternatively,  intangible  costs  and  raw  benefits  can  often  be  quantified  by  measuring  the 
consequences  of  not  making  the  investment  in  question;  money  saved  versus  the 
prospects  of  what  is  to  be  gained.  Thus,  in  the  final  analysis,  ROI  methodology  for 
intranet  initiatives  is  more  a  cost-benefits  analysis  (see  SectionV.F.2.b;  Activity  Based 
Costing  for  how  this  can  be  accomplished  in  the  form  of  process  changes  through  metrics 
analysis  based  on  the  lEEM). 

Standard  definitions  of  ROI  today  are  beginning  to  stray  from  their  original 
meaning  as  input  metrics  are  changing  to  accommodate  increasingly  dynamic 
environments  such  as  intranets.  Continuing  to  focus  on  cost  and  savings  is  an  operational 
management  contribution  to  “business  management”  but  it  doesn’t  give  satisfactory, 
complete  answers  about  the  contribution  of  intranet  portals  to  an  enterprise’s  value. 
These  answers  can  only  be  found  at  higher  levels  -  at  the  level  of  the  strategic 
management  and  economic  valuation.  Benefits  may  be  expressed  in  many  ways,  but  the 
key  is  to  express  them  in  understandable  terms  that  focus  on  value  by  matching  the 
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critical  business  requirement  issues  to  the  needs  and  inputs  of  knowledge  workers  and 
business  mangers  in  the  value  ereation  proeess  down  to  groups  of  specifie  metrics  that 
can  be  linked  and  measured  in  support  of  these  issues  and  needs  within  a  eoherent  and 
eomprehensive  framework.  The  lEEM  and  its  baseline  metrics  and  conversion  ratios  are 
designed  to  provide  this  approaeh  to  these  linkages  and  to  provide  insight  on  how  to  take 
eorreetive  aetion  upon  them.  The  ehoiee  executives  faee  is  not  whether  an  approaeh  like 
this  should  be  taken,  but  whieh  groups  of  metries  to  ehoose  and  how  to  proeeed  applying 
them  (see  Seetion  IV. C:  Periodic  Review). 

D,  APPLYING  WEB  ANALYTICS 

The  most  important  benefit  of  ealeulating  ROI  is  that  the  proeess  helps  determine 
whieh  metries  are  most  pertinent  to  a  partieular  business.  Prom  the  outset,  eompanies 
need  to  identify  and  stick  to  a  good  starting  point  when  taking  measurements  to  ensure 
what  is  being  measured  at  the  beginning  is  also  being  measured  at  the  end  in  a  systematie 
and  eomprehensive  manner.  This  is  particularly  true  in  measuring  the  performanee  and 
impaet  of  intranet  portals  sinee  so  little  has  been  previously  researehed  and  praeticed  in 
how  to  measure  them.  Unfortunately,  most  organizations  do  not  have  good  data  for  their 
intranet  ROI.  One  of  the  main  reasons  is  beeause  they  don’t  have  an  adequate  and 
coherent  eolleetion  of  “before”  data  from  eaeh  segment  of  their  intranet  operation 
proeess.  Very  few  eompanies  take  eomprehensive  snapshots  of  where  they  are  and 
equally  few  take  eomprehensive  snapshots  of  where  they  are  going  with  respect  to  overall 
strategic  and  value -based  objeetives. 

When  building  the  ROI  justifieation  for  intranet  investments,  specifie  metries 
must  be  predetermined  that  ean  be  used  to  properly  analyze  and  report  the  neeessary 
information.  These  metries  will  be  traeked  over  time  so  that  they  ean  be  mapped  to 
aetionable  solutions  related  to  bolstering  business  requirements  that  will  prove  out  the 
eorreetness  of  the  original  business  justifieation  of  the  projeet,  initiative,  or  enhaneement 
as  well  as  the  solution  itself.  In  addition,  the  ROI  methodology  and  set  of  assumptions 
must  be  used  in  a  consistent  format  to  better  enable  quantification  of  changes  in  growth 


69 


and  usage  patterns.  Rather  than  metries  portraying  what  happened,  a  dedicated, 
collective  effort  to  gather  this  information  and  analyze  it  helps  to  determine  what  to  do 
next  to  improve  performanee  (see  Chapter  VI,  Business  Intelligence  Team).  Thus  the 
proeess  of  measuring  and  fine  tuning  performance  intranet  portal  impacts  caused  by 
actions  based  on  metrics  analysis  is  an  economic  value  creator. 

1.  Pre-Determined  Metrics 

Enterprises  need  to  predetermine  the  metrics  they  will  collect  to  assess  their 
critical  business  value  objectives,  i.e.,  targeting  customer  loyalty,  partner  assessment, 
content  effectiveness,  channel  efficiency,  etc.  ROI  projections  for  portal  framework 
deployments,  though  useful  for  project  approval,  do  not  provide  insight  into  the  real  and 
actualized  value  derived  from  the  portal.  As  a  result,  planning  the  business  case  for 
portal  investments  will  require  predetermined  operational  metrics,  both  quantitative  and 
qualitative,  to  be  tracked  over  time.  Otherwise,  these  metrics  provide  no  meaningful 
insight  into  how  well  businesses  are  reaching  their  strategic  objectives. 

The  introduction  of  the  Intranet  Efficiency  and  Effectiveness  Model  (lEEM)  in 
this  paper  outlining  how  to  breakdown,  analyze  and  gauge  the  impact  of  changes  made  in 
support  of  critical  business  requirement  issues  all  the  way  down  to  specific  groups  of 
metrics  makes  it  possible  to  measure  the  effectiveness  of  an  organization’s  quality  drive 
toward  greater  value.  It  accounts  for  a  variety  of  factors  and  indicators  that  avoid  the 
problem  associated  with  isolating  the  specific  impact  of  any  one  of  those  factors  on  an 
overall  conversion  rate.  A  single  indicator  does  not  gauge  the  dial  measuring  overall 
success  but  a  collection  of  the  right  groups  of  indicators  and  the  metrics  to  measure  them 
can  achieve  this  task  over  time.  These  indicators  and  the  metrics  all  work  in  concert  to 
drive  an  accurate  conversion  rate.  The  lEEM  is  appropriately  broad  based  to  tractably 
gauge  and  subsequently  provide  enhancements  to  dozens  of  Web-related  initiatives  that 
have  an  impact  on  conversion  rate  because  the  constituents  in  the  model  take  into 
account,  for  example,  the  following  factors: 
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•  Navigation 

•  Site  layout 

•  Site  authoritativeness 

•  Prequalifieation  and  disposition  of  visitors 

•  Site  performance 

•  Scale 

•  Speed  to  fulfillment 

In  addition,  information  flood  and  false  alarms  are  essentially  prevented  by  defining 
multiple  metrics  to  describe  business  activities.  Consequently,  an  alert  is  triggered  only 
if  a  combination  of  metrics  shows  certain  behaviors.  Even  if  a  single  metric  tells  the 
whole  story,  it  is  better  to  have  two  metrics  linked  to  the  resultant  alert,  or  another  means, 
such  as  reviewing  earlier  analysis,  to  crosscheck  that  a  problem  really  exists  before 
alerting  anyone. 

2.  Intranet  Analytic  Omissions  and  Susceptibilities 

The  following  analytic  pitfalls  in  conducting  Intranet  ROI  are  outlined  to 
highlight  the  differences  and  subtleties  that  need  to  be  accounted  for  when  assessing  and 
measuring  the  value  added  from  intranet  processes. 

a.  Quantitative  and  Qualitative 

There  are  some  caveats  that  need  to  be  addressed  for  both  soft  and  hard 
metrics  with  respect  to  intranets.  Dotcoms  counted  the  number  of  “eyeballs”  driven  to  a 
virtual  storefront,  but  time  has  shown  that  the  quality  of  website  hits  and  a  site's  ability  to 
retain  customers,  known  as  stickiness  or  “recency”,  is  a  better  measure  of  ROI  and 
business  value  than  measuring  site  traffic.  Quantitative  metrics  in  and  of  themselves  can 
be  misleading  for  intranets  and  therefore  should  be  supplemented  with  e-surveys  to  fill  in 
qualitative  information  that  Web  logs  cannot  provide.  Because  some  critical  information 
gleaned  from  usage  data  or  analysis  of  intranet  Web  logs  is  inadequate  for  measuring 
ROI,  the  quality  of  the  knowledge  worker  experience  can  be  improved  by  implementing  a 
feedback  loop  consisting  of  regular  reviews  of  quantitative  and  qualitative  metrics.  For 
example,  a  page  can  have  high  traffic  because  the  content  is  uniquely  interesting,  it 
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represents  a  gateway  to  other  seetions,  or  knowledge  workers  are  stuek  in  a  frustrating 
eireular  navigation.  Similarly,  some  desirable  attributes,  sueh  as  ease  of  navigation, 
relevanee  of  seareh  results,  elarity  of  eontent  as  well  as  the  layout  of  the  site,  ean  only  be 
assessed  and  disambiguated  by  users.  Thus,  quantitative  metries  must  be  eorrelated  with 
qualitative  assessments  to  formulate  a  eomplete  pieture  of  the  user  experienee,  sueh 
foeused  surveys  whieh  ean  be  aeted  upon,  i.e.,  ehanges  to  the  layout,  links  and  visibility 
of  data. 


b.  Hit  Counts 

Quantitative  metries  like  popular  “hit  eounts”  are  most  eommonly  used  for 
intranets  today  beeause  they  are  readily  available  and  easier  to  ealeulate.  Onee  gleaned 
from  Web  logs  however,  they  present  a  number  of  ehallenges  to  deeision  makers  when 
used  for  intranets: 

•  The  number  of  hits  and  the  level  of  produetivity  ean  be  inversely  proportional. 
Organizations  moving  from  a  eomplex  statie  intranet  to  an  employee  portal  often  find 
that  the  number  of  hits  goes  down  beeause  less  surfing  is  needed  to  find  relevant 
information.  For  example,  the  portal  eould  do  the  surfing  for  the  user  based  on  a  speeifie 
user  profile,  thereby  bringing  the  information  direetly  to  the  user  via  another  eontent 
provider/department  portal. 

•  When  the  number  of  hits  is  used  to  justify  additional  modifieations  based  on 
traffie  volume,  a  low  number  ean  often  tell  a  better  story  than  a  high  number  (e.g.,  “We 
need  more  money  or  manpower  beeause  we’re  not  getting  hits  and  therefore  need  to 
provide  more  valuable  information.”).  This  ean  oeeur  as  a  result  of  faeilitating  the 
delivery  of  what  knowledge  workers  are  seeking  through  personalization,  i.e.,  plaeing 
speeifieally  frequented  links  on  the  desktop  homepage. 

•  Intranets  and  department  portals  generate  a  base  level  of  hits  even  if  they  are 
never  used,  due  to  auto-starting  (with  morning  boot-up  or  every  time  a  browser  window 
is  launehed)  and  multiple  eounting  (e.g.,  portals  generating  multiple  hits  for  eaeh  page  as 
“portlets”  are  rendered).  Portal  owners  should  ealeulate  the  base  level  of  hits  that  will  be 
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incurred  automatically  and  subtract  this  figure  from  total  hits  to  generate  the  number  of 
live  hits,  which  is  a  more  useful  measure  of  user  involvement. 

Hit  counts  out  of  context  are  of  limited  use.  Organizations  need  to  tie  hit  data  back  to  a 
role-  and  process-based  context,  e.g.,  matching  hits  with  profiles  to  determine  which  roles 
are  not  being  served  by  the  portal  and  which  functionality  is  most  used  for  each  role, 
determining  how  often  a  particular  task  is  accomplished  through  the  portal. 

Without  thorough  analysis,  even  the  simplest  metric  indicators,  like  hit 
counts,  can  be  misleading.  Failure  to  recognize  this  will  degrade  portal  performance  as 
wasted  time  and  effort  in  implementing  changes  based  on  incorreet  interpretations.  It 
may  be  something  small,  such  as  a  navigational  loop,  but  it  may  be  corporate -wide 
affecting  tens  of  thousands.  Thus,  even  the  most  straightforward  hard  metrics  should  not 
be  taken  for  granted,  but  meticulously  scrutinized  in  short  order.  Using  the  lEEM 
approaeh  over  time,  triggers  that  alert  analysts  to  these  potential  pitfalls  will  be  well 
instated  because  hit  counts  will  be  collected  with  a  variety  of  other  metrics,  such  as  e- 
surveys,  which  will  aid  in  identifying  pain  points.  Hard  and  soft  metrics  analyzed 
together  with  consideration  given  to  their  strengths  and  weaknesses  allows  analysts  to 
identify  incongruent  analysis.  Thus,  achieving  success  through  the  use  of  any 
performance  metric  will  depend  as  much  as  how  well  it  is  applied  as  it  does  on  when  it  is 
used  (see  Chapter  VI,  Business  Intelligence  Team  and  Section  IV.C:  Periodic  Review, 
respectively  for  how  to  realize  this). 

c.  Conversion  Rates 

Though  counter-intuitive,  since  so  many  factors  impact  on  the  conversion 
rate,  monitoring  a  conversion  rate  does  not  enable  businesses  to  determine  the  precise 
impact  of  any  one  factor.  Conversion  rate  measures  ostensibly  how  effective  the  site  is  at 
converting  its  visitors  to  browse,  download,  etc.  Hence,  it  needs  to  be  taken  into 
eonsideration  collectively  in  a  coherent  manner  that  eovers  a  variety  of  pertinent  factors 
from  each  process  segment,  which  the  lEEM  from  inception  is  designed  to  address. 
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Essentially,  a  portal  seeking  to  expand  its  reach  and  loyalty  should  deploy  initiatives  that 
convert  their  base  of  registered  users  and  first-time  visitors  into  loyal  repeat  customers: 
Focus  on  what  drives  customer  loyalty  and  higher  conversion  rates  will  follow.  The 
opposite  is  not  true  despite  much  attention  being  devoted  to  devising  new  conversion  rate 
techniques  to  manipulate  the  numbers  rather  than  what  factors  can  push  them  upwards. 

Although  efforts  at  cost  reductions  can  be  fairly  easily  applied  throughout 
a  firm,  efforts  aimed  at  increasing  effectiveness  generally  cannot,  unless  the  same  model 
and  set  of  metrics  are  applied  uniformly  across  all  key  portals  in  an  enterprise.  For 
example,  what  makes  one  employee  satisfied  or  productive  may  not  have  the 
same  impact  on  another  employee.  However  with  the  use  of  identical  metrics 
complemented  with  occasional  surveys,  data  anomalies  are  mitigated  with  periodic 
samplings  over  time  because  studies  from  samples  and  averages  based  on  the  same 
method  are  easier  to  compare.  In  spite  of  the  challenges  they  may  present,  it  is  important 
to  maintain  a  balanced  approach  while  pursuing  soft  benefits  —  such  as  customer 
satisfaction  and  understanding,  market  intelligence  and  knowledge  transfer  —  because 
they  contribute  to  middle  benefits  (derived  metrics)  —  such  as  speed-to-market  and 
loyalty  conversion  ratios  —  which  directly  impact  the  hard  numbers  that  build  a 
company's  bottom  line. 

d.  High-End  Knowledge  Workers 

The  best  ways  to  approach  this  exercise  is  by  letting  key  knowledge 
workers  express  what  is  useful  and  believable  or  not.  The  payback  is  great  since  even  a 
small  increase  in  the  effectiveness  of  a  firm's  most  critical  workers  can  impact  the  firm's 
bottom  line.  One  study  estimates  that  improving  the  performance  of  general  knowledge 
workers  adds  about  ten  times  more  to  the  bottom  line  than  facility  and  IT  cost  reductions 
combined  (Cantrell,  2001).  For  a  firm's  most  important,  "high-end"  knowledge  workers, 
this  ratio  is  bound  to  be  dramatically  higher,  cases  in  point  being  a  software  firm  or 
research  division.  Instead  of  employing  a  common  compromise  for  all,  an  enterprise 
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should  consider  solutions  more  oriented  toward  effectiveness  solutions  for  some,  and 
solutions  more  oriented  toward  efficiency  solutions  for  others.  Thus,  when  selecting 
enhancement  solutions,  analysts  and  BDMs  alike  should  bear  the  following  in  mind; 
Effectiveness  solutions  which  tend  to  be  more  intangible  and  soft  should  focus  on  high- 
end  knowledge  workers  (get  them  involved  in  the  feedbaek  loop)  and  efficiency  solutions 
should  target  employees  who  contribute  less  to  a  firm's  revenue,  sueh  as  administrative 
assistants. 


e.  Knowledge  Workers  ROI  Fallacy 

With  regard  to  different  types  of  enhaneements  for  different  levels  of 
knowledge  workers.  Capers-  Jones  estimates  productivity  gains  of  50-75%  are  possible 
(primarily  for  software  and  research  firms)  by  using  outstanding  programmers  and 
analysts  (Casper,  1986).  The  first  measurement  of  this  kind  was  the  Saekman's 
Experiment  in  whieh  large  individual  differences  were  found  to  exist  between 
programmers  (Sackman  1968).  Another  study  of  this  kind  conducted  20  years  later, 
known  as  Demareo's  Coding  Wars,  found  similar  results  but  not  as  dramatic  (Demarco, 
1999,  p.  27).  Table  4  below  shows  their  research  results  between  more  and  less 
proficient  programmers  in  an  organization  given  the  same  amount  of  time  to  program: 


Saekman’s  Results 

Demarco's  Results 

Debug  Hours  : 

18-1 

Best  people  will  outperform  the  worst  by  10:1. 

Code  Hours  : 

15-1 

Best  performer  will  be  2.5  times  better  than  the  median. 

Program  Size  : 

6-1 

The  top  1/2  will  outperform  the  bottom  1/2  by  2:1. 

Run  Time  : 

13-1 

Table  4,  Programmer  Productivity  Results 


A  cursory  conclusion  to  this  is  that  organizations  should  focus  their  portal 

efforts  on  accommodating  their  high-end  knowledge  workers  with  all  the  means 

neeessary  to  do  their  jobs  better.  Although  this  is  not  entirely  incorreet,  it  overlooks  two 

important  faetors  with  respeet  to  ROE  1)  less  skilled  programmers  do  not  get  paid  10 

times  less  salary;  and  2)  there  are  normally  fewer  high-end  programmers  than  the  lower- 
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end  knowledge  programmers  that  work  on  any  given  projeet  (due  to  a  variety  of 
eireumstanees  sueh  as  promotions  over  time  of  the  more  experieneed  programmer  to 
management  positions).  Thus,  eorporations  need  to  exereise  eaution  when  alloeating 
resourees  and  prioritizing  portal  enhaneements.  The  paybaek  may  be  greater  for  features 
or  ehanges  that  affeet  a  wider  body  of  knowledge  workers  and  programmers  who  are 
eonsidered  median  or  lower-end  than  for  a  smaller  high-end  group.  This  of  eourse  will 
be  deeided  on  a  ease-by-ease,  or  portal-by-portal  basis,  and  is  why  dynamieally 
eonstrueted  portals  designed  to  meet  the  needs  of  like  users  is  a  powerful  new 
development  in  the  IT  world.  All  the  same,  analysts  and  BDMs  must  keep  in  mind 
knowledge  worker  eeonomies  of  seale  when  pareeling  resourees  to  enhanee  produetivity. 

f  Knowledge  Workers  and  Reuse 

The  eombined  and  logieal  approaeh  of  model  based  seleeted  hard  and  soft 
metries  ean  lead  to  better  identifying  and  understanding  what  knowledge  workers  are 
doing  with  what  they  discover  in  intranet  portals  as  well  as  quantifiable  and  favorable 
ROT  For  example,  as  a  result  of  hard  and  soft  metrie  analysis,  ROI  may  appear  in  some 
unexpeeted  plaees  sueh  as  reuse  of  software  eode.  Software  reuse  is  a  very  measurable 
and  desirable  as  it  allows  eost  per  funetion  delivered  to  be  dramatieally  redueed.  “For 
instanee,  a  1,200-member  IT  team  at  a  Cleveland-based  finaneial  firm  eut  its  average 
projeet  turnaround  time  by  an  astonishing  45%  after  it  diseovered  software  in  a 
development  team’s  portal  that  would  suffiee  for  other  internal  projeets.  On  average,  a 
single  software  eomponent  took  200  hours  to  design,  at  a  ehargebaek  rate  of  $74  per 
hour,  or  $14,800  per  eomponent.  When  one  eomponent  was  reused  in  eight  different 
projeets,  it  saved  the  eompany  more  than  $100,000”  (Frakes,  2003,  p.  31).  Auditing  Web 
log  files  alone  would  not  have  eaptured  this,  portal  quality  traeking  eomplimented  with 
user  feedbaek  resulted  in  spotting  and  analyzing  ROI  returns  that  would  have  normally 
gone  unreeognized  and  unaeeounted. 
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With  respect  to  reuse,  a  promise  of  software  and  intranets  is  lower  costs  or 
at  least  getting  more  “bang  for  the  buck”.  Focusing  on  code  improvement  through  portal 
optimization  is  highly  desirable  because  it  can  result  in  both  higher  reliability  and  faster 
time  to  market.  However,  software  reuse  until  recently  has  been  noticeably  missing. 
Although  there  is  promise  of  turning  this  phenomena  around  with  the  advent  of  C#  within 
.NET  and  free  Linux  libraries  online,  many  knowledge  workers  do  not  even  know  the 
code  they  are  writing  has  already  been  written  or  something  very  similar  to  it  can  be 
modified  in  its  place  -  even  when  it  exists  on  their  own  intranet.  If  made  an  objective, 
properly  configured  portals  will  facilitate  software  reuse  and,  more  important,  knowledge 
reuse,  by  exposing  it  and  providing  such  things  as  references  to  design  documents  used  to 
implement  the  module  should  it  be  included  in  other  software  design  templates.  Using 
the  same  metrics  groups  over  and  over  again  across  the  enterprise  to  gauge  performance 
is  itself  also  a  form  of  reuse  provided  by  lEEM  domain  analysis  and  metrics  baseline  (the 
key  to  reusable  software  is  captured  in  domain  analysis  in  that  it  stresses  the  reusability 
of  analysis  and  design,  not  code).  Collectively,  these  are  example  of  how  a  best  practice 
can  become  a  business  rule  whereby  virtually  all  code  must  be  made  accessible  along 
with  clear  understandable  documentation.  If  a  business  rule  is  not  possible,  then  an 
incentive  program  can  be  devised  to  reward  and  recognize  portals  whose  code  or 
documentation  is  most  widely  and  highly  used  by  other  co-knowledge  workers. 

E.  OBJECTIVE  OF  ROIMI 

Used  properly,  Web  analytics  can  provide  significant  returns  in  optimizing  portal 
configurations  and  capabilities  if  based  on  a  model  that  accounts  for  critical  business 
requirements,  measured  consistently  and  periodically  to  determine  where  refinements  are 
needed  in  order  to  keep  in  step  with  the  dynamic  needs  of  users.  This  is  exactly  why  the 
lEEM  has  been  created  so  these  refinements  can  be  conducted  in  a  logical  and  coherent 
manner  as  they  impact  on  critical  business  requirements,  namely  to  increase  value. 
Figure  12  below  provides  a  theoretical  illustration  why  a  portal  optimized  using  focused 
metrics  is  more  effective  and  inevitably  more  productive  than  one  that  does  not. 
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Slope  of  Decision  Points  to  take  Informed  Action 
Portal  that  undergoes  regular  optimization 
Portal  that  undergoes  little  or  no  optimization 


Figure  12,  Improving  and  Reducing  the  Decision  Points 


Given  the  time  it  takes  to  find  information  A  in  time  t‘  in  an  a  portal  that  is  not 
optimized,  a  portal  that  is  regularly  measured  to  take  into  aeeount  user  aetivity  and 
behavior  is  more  optimized  to  meet  users’  needs  and,  therefore  will  render  more  desired 
relevant  information  ^4 +5  in  the  same  time  period  (point  7  versus  point  6).  On  the  other 
hand,  if  time  is  of  the  essence,  information  A  can  be  found  in  less  time  (point  5),  thereby 
allowing  the  user  more  time  to  consider  the  information  or  to  come  to  a  decision  sooner 
(point  6  versus  point  2).  Either  way  the  optimized  portal  will  provide  as  much  relevant 
information  in  a  shorter  time  period  or  more  information  in  the  same  time  period.  The 
result  is  either  a  business  decision  sooner  or  a  more  informed  one  respectfully.  Another 
phenomenon  is  the  near  exponential  affect  of  this  event  over  time  with  regard  to  an 
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increase  in  relevant  information  and  the  deeision  process  itself.  Though  time  is 
truncated,  it  may  not  always  be  exponential  as  the  discovery  of  relevant  information  can 
be  as  a  result  of  better  association  and  placement  of  documents  and  information  based  on 
user  behavior.  Moreover,  time  spent  deliberating  to  reach  a  decision  may  also  be 
truncated  by  a  significant  margin  in  some  cases  since  more  relevant  information  is  found 
providing  clarity  on  what  courses  of  actions  are  more  likely  than  others. 


At  any  given  point  in  time,  there  will  be  as  more  information  found  in  a  portal 
consistently  using  metric  analytics  based  on  the  lEEM,  resulting  in  more  informed 
business  decisions  being  reached  in  less  time.  In  theory,  this  is  how  intranet  portals 
facilitate  speed  to  market.  Table  5  below,  lEEM  Analysis  Impact  on  Time  to  Reach 
Decision,  is  an  extension  of  Figure  12  above,  showing  the  differenee  between  intranet 
portals  without  the  guidanee  of  lEEM  metrics  versus  those  managed  with  them; 


Best 

Portal  Type 

Time  Info 
Gathered 

Amount  Info 
Colleeted 

Time  to  Reach 
Informed 
Deeision 

PRO/ 

CON 

1 

Portals  using 
metrics 

Tl/2 

A 

<T1 

Best  with 
little  time 

2 

Portals  using 
metrics 

T1 

A+B 

<t2 

3 

Portals  using 
metrics 

T2 

A+B+C+D 

<T4  (=t3) 

Best  with 
more  time 

4 

Portals  w/no 
metrics 

T1 

A 

T2 

Worst 
with  little 
time 

5 

Portals  w/no 
metrics 

T2 

A+B 

T4 

Table  5,  lEEM  Analysis  Impact  on  Time  to  Reach  Decision 


There  are  many  variables  that  eould  skew  the  results  for  time  to  reaeh  a  business 
deeision  to  be  consistent  (i.e.,  individual  skill  sets  and  experienee  ean  vary  greatly). 
However,  if  improvements  are  made  in  the  other  metrie  areas  outlined  by  the  lEEM  in 
Chapter  III,  Conceptual  Framework,  they  will  collectively  help  to  minimize  the  time  to 
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locate  desired  information  and  the  resulting  deeision  reached.  Time  to  locate  is  a  classic 
example  that  helps  put  the  figure  and  table  above  into  perspective.  Oeeasionally,  users 
may  find  what  they  are  looking  for  sooner,  but  will  also  continue  to  look  for  long  periods 
of  time  (perhaps  as  much  as  the  approximate  50%  of  their  time  as  they  do  now)  because 
they  are  finding  more  of  what  they  are  seeking.  Regardless,  the  time  factor  is  reduced 
with  respect  to  finding  what  is  sought  or  considered  desirable:  If  people  still  spend  50% 
of  their  time  looking  for  information,  they  should  have  more  pertinent  information  than 
before  in  the  same  amount  of  time  -  which  should  lead  to  better  deeision  making  and 
ultimately  more  effectiveness.  Time  to  loeate  is  an  ejjiciency  metrie  that  is  affected  by 
increases  in  effectiveness  elsewhere.  The  efficieney  metric  of  time  to  locate  in  turn 
affects  effectiveness  across  the  board  because  users  will  either  have  or  ean  do  more  in 
less  time.  Thus,  this  metrie  area  is  an  example  of  how  effectiveness  affects  efficiency 
and  then  how  this  efficiency  increase  in  turn  improves  effeetiveness. 

F.  DERIVING  AND  EMPLOYING  A  COMMON  UNIT  OF  ANALYSIS 

Time  is  money  and  the  unit  of  analysis  most  appropriate  to  measure  the  ROI 
impact  of  the  lEEM  metrics  based  analytics  is  indeed  time.  Although  it  may  be 
challenging  to  put  exact  figures  on  the  impact  of  every  intranet  portal  project,  placing  a 
cost-benefit  ROI  on  the  worth  of  applying  the  lEEM  and  metrics  baseline  analytics  across 
a  large  enterprise  is  attainable.  The  key  assumption  here  is  that  a  corporation  would 
assume  the  lEEM’s  supporting  metries  to  serve  as  a  baseline  to  perform  analyties  only 
when  it  already  has  an  intranet  (i.e.,  sunk  cost  as  part  of  doing  business)  and  it  believes 
there  is  room  for  eontinuous  optimization  in  it  to  increase  value.  The  ROI  conducted  is 
essentially  a  eost-benefit  between  the  time  needed  to  invest  in  applying  and  acting  upon 
the  results  of  the  analytics  (i.e.,  eosts  to  hire  an  outside  analyst  or  establish  an  in-house 
team  to  conduct  the  analyses)  and  the  results  in  time  reduetions  due  to  subsequent 
ehanges  introduced  by  this  analysis  proeess,  i.e.,  shorter  completion  time  of  a  series  of 
business  related  tasks  (sub-corporate  level)  and  speed  to  market  or  completion  of  a 
project  (corporate  level).  The  inereases  in  quality,  ereativity  and  knowledge  worker  or 
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customer  satisfaction  will  be  strongly  implicit  but  each  falls  short  of  readily  breaking 
down  into  precise  units  of  time,  only  approximations  of  it. 

Time  savings  is  sought  through  the  optimization  of  intranet  portals  which  are 
directly  correlated  to  impacts  on  both  efficiency  and  effectiveness  gains  in  support  of 
business  value  objectives.  As  outlined  in  Section  V.E,  Objective  of  ROIMI,  this  can  be 
accomplished  only  if  the  same  groups  of  metrics  are  applied  periodically  to  obtain  before 
and  after  results.  Otherwise,  the  comparison  between  the  two  sets  of  data  collapses  and 
taking  subsequent  actions  to  enhance  performance  related  to  the  results  of  specifically 
crafted  groupings  of  metrics  are  less  certain  and  valid.  The  reductions  in  time  must  be 
compared  against  previous  baseline  measurements  to  gauge  the  extent  of  performance 
improvements  (see  Section  IV. C:  Periodic  Review  and  Figure  10;  Implementation 
Scenarios  for  Applying  a  Baseline  Metrics). 

Practitioners  who  redesign  business  processes  require  a  method  for  determining 
how  much  their  process  design  decisions  will  impact  performance  (El  Sawy,  2001). 
During  the  lifetime  of  this  approach  other  combinations  of  metrics  may  be  applied  which 
are  deemed  more  precise,  but  these  should  only  take  place  after  at  least  two  to  three 
periodic  measurements  have  already  been  fully  conducted  and  analyzed  to  mitigate 
anomalies  and  correct  errors.  This  method  thereby  provides  a  convenient  way  to  estimate 
the  returns  that  alternative  process  design  changes  can  generate.  Thus,  the  lEEM 
framework  portends  to  resolve  the  long-standing  problem  in  the  IT  community  of 
determining  the  IT  initiative  impacts  on  a  large  number  of  processes  at  precise  enough 
levels  of  the  entire  find  experience  to  benefit  managers  who  must  implement  changes  at 
the  tactical  level  and  still  link  them  to  strategic  business  objectives.  The  rest  of  this 
chapter  explains  how. 
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1,  Entropy  Concept 


The  credibility  and  applicability  of  this  conjecture  are  significantly  fortified  by 
associating  the  metric  performance  indicators  to  legitimate  and  logical  granular  unit  of 
measurement.  The  technique  which  meets  the  requirement  of  determining  the  output  of 
time  savings  as  well  as  enabling  its  “operationalization”  of  this  theory  is  the  Knowledge 
Value  Added  (KVA)  theory  which  offers  a  practical  method  for  estimating  the  value 
added  by  IT  via  theories  rooted  in  assumptions  derived  from  entropy  in  complexity  and 
thermodynamics  concepts: 

The  changes  organizational  processes  make  in  the  structure  of  inputs  to 
produce  outputs  can  be  described  in  a  common  way  in  terms  of  the 
entropy  concept.  The  concept  of  entropy  is  defined  as  a  measure  of  the 
degree  of  disorder  or  change  in  a  system.  In  the  context  of  business 
processes  it  can  be  used  as  a  surrogate  for  the  amount  of  changes  that  a 
process  makes  to  inputs  to  produce  attendant  outputs.  These  process- 
induced  changes  can  be  measured  in  terms  of  the  equivalent  corresponding 
changes  in  entropy  (Housel  and  Kanevsky,  1995). 

Within  the  framework  of  thermodynamics,  a  fundamental  parallelism  between 
transformation  of  substances  and  information  processing  has  been  established  (Li  and 
Vitanyi,  1993).  If  a  substance  is  transformed  from  state  a  to  state  b,  then  the  difference 
of  the  entropies,  i.e.,  AE=E(b)-E(a),  is  proportional  to  the  amount  of  thermodynamic 
work  required  for  the  change  (Housel  et  al.,  2001,  p.  1 1).  In  other  words,  application  of 
knowledge  is  determinant  of  value.  A  process  must  enact  some  change  upon  inputs  to 
produce  an  output  of  value.  Therefore,  change  can  create  value  and  knowledge  is 
proportional  to  value. 


As  theorized  by  Housel,  Rodgers,  El  Sawy  and  Zhong,  by  extending  this 
conceptualization  of  the  relationship  between  complexity  and  entropy  in 
the  organizational  context,  conditional  complexity  can  be  viewed  as  the 
shortest  description  of  the  process,  i.e.,  effectively,  the  productivity  of  the 
process.  Further,  a  change  in  entropy  when  state  a  is  transformed  into 
state  b  depends  only  on  a  and  b  and  does  not  depend  on  process  P.  This 
means  that,  by  definition,  any  process  P  that  changes  a  into  b  introduces 
the  same  change  in  entropy  or,  in  an  organizational  context,  adds  the 
same  value.  It  is  reasonable  then  to  assume  that  the  minimal  set  of 
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instructions  to  change  a  into  b,  via  process  P  refleets  the  corresponding 
change  in  entropy  given  the  current  state  of  process  P.  In  other  words, 
the  length  of  the  shortest  description  of  the  change  provides  an  acceptable 
approximation  of  the  change  in  entropy  given  the  eurrent  state  of  the 
process.  This  beeomes  eritical  in  reeognizing  that  estimations  of  ehanges 
in  entropy  can  only  ever  be  approximations  (Housel  et  al..,  2001,  pp.  12- 
14). 

Thus,  given  that  the  estimates  are  derived  using  the  common  theoretical  framework  in 
lEEM,  it  follows  that  a  simple  eorrelation  between  process  and  outcome  leads  to 
reasonable  approximation  of  the  reliability  of  the  estimates. 


At  a  given  point  in  time,  a  company’s  total  process  outputs  produee  its  revenue. 
It  follows  that  the  proeedural  knowledge  required  to  produee  those  outputs  is  a  surrogate 
for  the  revenue.  Eurther,  if  this  proeedural  knowledge,  which  is  distributed  among  people 
and  IT,  can  be  deseribed  in  common  units,  then  it  is  possible  to  allocate  eorporate 
revenue  to  these  units  of  knowledge.  This  would  allow  establishment  of  a  common  price 
per  unit  of  procedural  knowledge.  It  follows  that  price  per  unit  of  proeedural  knowledge 
is  a  surrogate  for  price  per  unit  of  common  output.  This  formulation  allows  a  direct 

®  Housel  et  al.,  2001,  p.  14. 
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linkage  between  eorporate  revenue  and  the  proeedural  knowledge  distributed  among  the 
people  and  IT  used  to  produee  the  revenue.  Henee  it  would  be  possible  to  alloeate  the 
proportionate  revenue  produeed  by  the  procedural  knowledge  in  business  processes 
including  the  knowledge  contained  in  the  supporting  IT.  This  approach  establishes  the 
relationship  between  cost  and  resulting  productivity  (Housel  et  ah,  2001,  p.  13). 

2.  Surrogate  for  Value 

The  relationship  between  change  in  entropy  and  value  added,  while  fundamental, 
does  not  provide  a  practical  way  to  calculate  the  value-added  by  organizational  processes, 
i.e.,  the  entropy  increment  (Housel  et  ah,  2001,  p.  13).  The  time  it  takes  the  average 
learner  to  acquire  the  procedural  knowledge  required  to  produce  a  process  output 
provides  one  practical  surrogate  for  the  corresponding  changes  in  entropy  (see  Figure  13, 
Procedural  Knowledge  is  Proportionate  to  Change).  This  framework  can  be  applied  to 
the  context  of  organizational  processes:  Processes  with  predetermined  outputs  may  be 
described  in  terms  of  the  amount  of  time  it  takes  the  average  knowledge  worker  to 
produce  those  outputs  they  normally  do  to  complete  a  business  task.  It  follows,  that  the 
procedural  knowledge  used  to  produce  the  attendant  outputs  may  be  viewed  as  a 
surrogate  for  the  process  outputs. 

The  impact  of  procedural  knowledge  can  be  viewed  in  two  respects  in  the  context 
of  lEEM  and  portal  optimization:  one  is  the  reduction  of  knowledge  worker  task 
completion  time  to  exercise  the  same  level  of  procedural  knowledge  to  create  equally  if 
not  improved  outputs  (this  respect  is  how  KVA  can  be  applied  and  is  analogous  to 
learning  time  to  conduct  a  task  that  has  been  assigned  a  percentage  of  revenue  generation 
for  a  known  outcome  whose  revenue  returned  can  be  estimated);  and  other  is  in  the 
context  of  cost  benefit  showing  how  more  tasks  (ergo  more  expected  outcome)  can  be 
accomplished  in  the  same  period  of  time  or,  alternatively,  how  the  same  expected 
outcome  can  be  achieved  in  less  time  (this  respect  is  how  Activity  Based  Costing  can  be 
applied,  see  Section  E.2.b:  Activity  Based  Costing  and  lEEM). 
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In  the  knowledge  worker  task  eompletion  time  approaeh,  the  total  amount  of  task 
eompleting  time  required  to  business-related  outputs  (i.e.,  research,  code  or  service)  is  a 
surrogate  for  the  revenue  derived  from  a  firm’s  outputs  during  a  given  sample  period. 
The  outputs  of  all  the  company  processes  used  to  generate  this  revenue,  at  a  given  point 
in  time,  can  be  described  in  common  units  of  task  completion  time.  It  follows  that  “price 
per  unit  of  output,”  or  its  surrogate  “price  per  unit  of  knowledge,”  (which  is  derived  by 
dividing  company  revenue  by  the  total  number  of  units  of  knowledge)  is  a  constant. 
However,  the  cost  per  unit  of  knowledge  will  vary  depending  on  the  cost  of  the 
knowledge  resources  (e.g.,  people  and  technology)  used  to  produce  a  process  output 
(Housel  et  ah,  2001,  p.  16). 

One  task  for  example  in  which  the  lEEM  can  be  calculated  with  KVA  is  through 
greater  awareness  of  content  which  allows  for  greater  opportunities  for  such  things  as 
software  reuse  or  the  reduced  time  to  complete  any  standard  business  routine.  Even  more 
important  than  reuse  or  number  of  lines  of  reduced  code  per  se  is  the  derived  benefits 
portals  provide  in  improving  the  chances  for  faster  development  and  time  to  market.  Eor 
example,  many  additional  products  are  produced  as  a  result  of  discovering  code  or  useful 
information  that  would  have  been  otherwise  inconceivable.  Thus,  the  usability  of 
valuable  documents  and  artifacts  previously  unrecognized  creates  competitive  advantage. 
Since  the  over-arching  objective  of  an  intranet  is  time  reduction  to  complete  all  tasks, 
enhancements  to  process  design  as  a  result  of  lEEM  analysis  to  optimizing  portals  for 
knowledge  worker  productivity  needs  is  therefore  the  most  crucial  issue  in  facilitating 
and  maintaining  the  highest  returns  possible  for  an  intranet.  The  advantages  of 
employing  the  Knowledge  Value  Added  and  Activity  Based  Costing  methodologies  in 
conjunction  with  lEEM  are  that,  while  grounded  in  a  theoretical  framework,  they  can  be 
applied  practically  to  obtain  estimates  grounded  in  common  units  and  that  these  units 
can  be  used  as  a  surrogate  for  value  as  well  as  compared  to  each  other  to  ensure 


85 


opportunities  to  increase  value  are  not  overlooked.  How  these  two  approaches  to 
estimating  ROI  compliment  each  other  are  addressed  in  more  detail  in  the  remainder  of 
this  section. 


a.  Knowledge  Value  Added  and  lEEM 

As  changes  in  process  design  may  be  the  most  crucial  issue  in  predicting 
and  maintaining  the  highest  ROIMI  to  best  leverage  knowledge  embedded  practices 
within  and  across  intranet  portals,  good  old-fashioned  ROI  modeling  does  not  permit 
enough  time  to  develop  the  business  case  for  either  the  metrics  or  the  changes  suggested 
by  their  results.  Therefore,  calculating  returns  using  the  KVA  approach  in  conjunction 
with  the  lEEM  works  like  an  investment-portfolio  approach:  the  changes  made  to  process 
are  thought  to  payoff  and  improve  value,  but  a  period  of  time  is  needed  to  collect  the  data 
periodically  and  analyze  it  before  value  realizations  can  occur,  generally  in  the  form  of 
recognizing  greater  savings  in  time  or,  conversely,  more  productivity-related  activity  in 
less  time  as  a  result  of  the  application  of  and  enhancements  to  the  exercise  of  procedural 
knowledge. 


The  essence  of  KVA  is  that  it  takes  knowledge  utilized  in  corporate  core 
processes  and  translates  it  into  a  numerical  form  that  allows  allocation  of  revenue  in 
proportion  to  the  value-added  by  the  knowledge  as  well  as  the  cost  to  use  that  knowledge 
(Baskerville,  1999,  pp.  20-21).  Tracking  the  conversion  of  knowledge  into  value,  while 
measuring  its  bottom  line  impacts,  enables  managers  to  increase  the  productivity  of  these 
critical  assets  —  namely  in  this  study,  the  crucial  process  activities  that  take  place  in 
portals  that  drive  productivity. 

Although  the  KVA  methodology  can  be  applied  at  any  level  in  a  company, 
it  takes  on  significant  value  when  applied  at  the  enterprise  level.  A  form  of  the  KVA 
methodology  when  used  within  the  lEEM  framework  allows  a  business  intelligence  team 
to  iteratively  generate  estimates  of  return  on  portal  information  system  related 
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initiative/process  changes  as  they  test/tweak  various  proeess  designs  modifieations  within 
and  aeross  a  variety  of  portals.  In  this  manner,  eompetitive  advantages  of  this  faeulty, 
i.e.,  ehanges  brought  about  by  the  induetion  and  deduetion  of  metrie  results  based  off  the 
lEEM,  ean  be  reflected  in  eontributions  to  the  eompany  bottom  line  over  time,  not 
overnight. 


ROIMI  essentially  boils  down  to  a  delta  in  time  savings  as  the  time 
devoted  to  applying,  analyzing  and  taking  subsequent  aetions  based  on  the  lEEM  metrie 
eonversion  rates  results  in  a  net  gain  in  time  —  or  what  ean  be  aeeomplished  in  that  same 
period  of  time  —  in  the  work  of  all  workers  within  the  enterprise.  By  extension,  this  gain 
in  time  ean  be  plausibly  extrapolated  into  a  gain  in  value  (see  Appendix  D:  Time  to 
Locate,  for  explanation  of  how  this  is  taken  into  aeeount  by  the  lEEM).  One  high  level 
method  of  expressing  the  time  in  terms  of  savings,  whieh  is  easier  and  more  justifiable 
for  some  parties  as  opposed  to  value  ereated,  ean  be  done  by  representing  the  delta  in  the 
time  devoted  by  the  small  group  of  people  responsible  for  making  all  the  proeess  ehanges 
to  portals  eompared  to  aggregate  statisties  at  eaeh  portal  previously  and  then  to  the 
enterprise  as  a  whole  prior  to  the  ehanges  taking  affeet. 

Eor  example  if  Time  gained  (zl7)  as  a  result  of  time  invested  into  applying 
and  aeting  on  lEEM  metries,  then  AT  equals  the  amount  of  time  gained  by  all  employees 
from  one  measured  period  of  time  to  the  next  A{eT)=  eTi-eT2  minus  the  total  time  spent 
by  the  business  intelligenee  team  {bitT)  in  applying  and  analyzing  a  baseline  of  metries 
uniformly  aeross  the  enterprise,  effeetively  affeeting  a  wide  bodies  of  knowledge  workers 
(as  well  as  others  eausality  speaking).  A  simple  expression  of  this  is  zir=  A{eT)  -  (bitT). 
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An  example  of  how  this  equation  may  play  out  in  a  large  corporation  is: 

AT  =  A{eT)  -  (bitT) 

A{eT)  =  eTj  -  eTi 

First  30  Period  Second  30  Period 

Prior-  metric  changes  Post-  metric  changes 
eT I  —  Total  No.  K-Workers  x  Total  Hours  On-Line 

5000  X  4  hr/day  x  20  days  =  400,000  hours 

eJj  =  Total  No.  K-Workers  x  Total  Hours  On-Line 
5,000  X  3.8  hr/day  x  20  days  =  380,000  hours 

A{eT)  =  400,000  -  380,000  hours 
=  20,000  hours 

AT  =  A{eT)  -  (bitT) 

20,000  hours 

(bitT)  —  Total  No.  in  BIT  x  Total  Hours  Worked 
10  X  4hrs/day  x  20  days 
=  800  hours 

AT  =  20,000  hours  -  800  hours 

=  19,200  hours  gained  during  second  period 


In  this  example,  there  is  a  1:24  ratio  in  time  invested  by  BIT  and  time 
saved  to  other  knowledge  worker  company  wide  as  a  result  of  their  process  changes. 
This  assumes  that  all  portals  are  being  measured  and  that  most  employees  use  their 
intranet  to  conduct  business.  Although  this  ratio  difference  increases  with  the  size  of  an 
organization  and  its  intranet,  it  will,  conversely,  diminish  at  some  point  and  go  the  other 
directions  as  these  employee  numbers  and  intranet  sizes  decrease.  Even  if  one  quarter  of 
the  estimate  above  is  realized  during  a  30  day  period,  this  enhancement  to  the  exercise  of 
procedural  knowledge  would  still  result  in  a  theoretical  saving  of  over  57,000  hours  per 
year  (.25  x  19,200  x  12months).  It  is  these  kinds  of  numbers  that  will  get  attention  and 
prove  why  focusing  on  changes  to  process  is  the  key  to  higher  returns. 


These  results  can  be  expressed  as  magnitude  percentage  estimates, 
something  mangers  at  multiple  levels  may  be  more  willing  to  share  and  allow  further 
interventions  since  specific  dollar  amounts  resulting  from  the  interventions  may  be 


hidden  from  competitors  while  being  able  to  share  the  general  results  of  these 
interventions.  Another  tactic  in  presenting  performance  metrics  and  ROI  in  this  case  is 
that  it  may  be  wiser  to  use  a  range  of  numbers  rather  than  a  single  target  point.  Low-end 
numbers  enable  management  to  make  a  decision  based  on  conservative  projections. 
High-end  numbers  dangle  the  prospect  of  bigger  potential  gains  in  front  of  them, 
informing  them  in  any  event  of  the  significant  scale  these  changes  can  make  (see 
Appendix  F:  Spreadsheet  of  ROI  Returns  for  examples  of  both  ROI  magnitude 
percentage  estimates  and  target  range  of  numbers).  A  clearer  understanding  of  the 
potential  return  should  encouraging  BDMs  to  take  risks  (with  such  investments  of  time 
invested  by  a  few  knowledge  workers  to  reduce  time  on  task  for  many  knowledge 
workers)  until  they  have  an  accurate  way  of  gauging  potential  value-creating  benefits  in 
harder  numbers.  In  summary,  this  method  provides  a  means  -  as  well  as  a  rational 
justification  —  in  which  to  calculate  ROI  for  the  metrics  investment  (time  saved  and 
better  used  by  enhancements  to  procedural  knowledge  within  processes)  with  a  common 
unit  of  analysis:  time  to  exercise  procedural  knowledge  to  produce  an  expected  outcome, 
which  can  always  be  translated  into  money  saved  or  earned  as  well  as  anecdotally 
accepted  as  boosting  competitive  advantage,  i.e.,  time  to  market. 

This  example  also  helps  to  explain  why  a  form  of  the  KVA  methodology 
can  be  made  to  be  applicable.  For  one  its  basis  in  entropy  concepts  takes  into  account 
change  or  the  process  that  enables  this  and  that  these  changes/outputs  are  the  thing  of 
value  since  customers  pay  for  the  output  of  the  corporation  at  any  given  point  in  time. 
Simply  put,  it  allows  allocation  of  revenue  to  the  corporate  and  sub-corporate  levels  and 
it  allows  description  of  all  process  outputs  in  common  units.  These  common  units  are 
proportionate  to  revenue  and  thus  revenue  can  be  allocated  to  these  units  allowing  the 
generation  of  a  numerator  for  an  ROI  ratio  that  is  not  based  on  any  form  of  cost 
including,  cost  savings,  cost  avoidance,  investment  cost,  etc.^ 


^  Housel,  email  to  Grant  Jacoby  on  19  November  2003. 
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In  the  case  of  the  example  above,  KVA  could  be  applied  to  the  lEEM  for 
the  time  it  takes  someone  to  do  their  job  without  optimizing  changes  being  made  to  the 
process  compared  to  someone  who  has  the  same  tasks  but  with  the  optimized  settings.  It 
would  be  something  akin  to  learning  or  doing  a  job  and  not  having  to  learn  or  do  as  much 
to  in  order  to  accomplish  the  same  requirement  as  a  result  of  the  application  of 
knowledge  exercised  by  the  analysis  team  in  the  form  of  the  changes  they  make  to  crucial 
productivity  processes  of  portal  intranets. 

In  other  words,  the  time  to  exercise  knowledge  based  off  the  metrics  and 
applied  to  impactful  productivity  process  changes  of  portals  can  sow  even  greater 
knowledge  gains  into  the  enterprise  at  large,  i.e.,  the  1:24  time  savings  ratio  given  above. 
With  regard  to  efficiency  and  effectiveness,  time  is  saved  and  value  created  by  virtue  of 
people  being  able  to  do  more  in  less  time  —  again,  be  it  learning  or  doing. 
Concomitantly,  since  these  process  changes  ultimately  go  across  multiple  portals  there  is 
an  effectiveness  gain  as  well,  creating  value. 

b.  Activity  Based  Costing  and  lEEM 

Another  and  complimenting  technique  that  can  be  used  to  measure  ROIMI 
for  lEEM  based  metrics  is  Activity  Based  Costing.  ABC  is  a  popular  cost-based 
approach  because  finding  the  true  costs  of  process  activities  are  clearly  useful  in 
evaluating  them  (Johnson  and  Kaplan  1987).  Despite  the  fact  that  the  ABC  technique 
appears  to  be  a  very  suitable  managerial  tool  for  e-business,  widely-known  published 
reports  about  its  use  in  intranet  or  internet-related  economies  do  not  currently  exist. 
Nevertheless,  applications  of  ABC  to  measuring  the  impacts  of  IT  assume  that  any 
cost/time  saved  or  processes  simplified  (and  thus  costs  or  time  reduced)  by  the  IT 
(namely  in  this  case  the  corrective  actions  taken  after  analysis  of  lEEM  metrics  and 
conversion  rates)  are  a  direct  reflection  of  its  value.  This  assumption  holds  true  in  cases 
where  lEEM  metrics  analysis  is  applied  causing  reductions  in  cost/time  while  process 
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outputs  remain  constant  or  increase.  Thus,  the  applicability  and  merits  of  ABC  to  lEEM 
related  ROIMI  warrants  further  examination. 

Criticisms  of  ABC  need  to  be  kept  in  mind  and  overcome  if  it  is  to  have 
credibility  in  assigning  value  to  process  changes  brought  about  by  analysis  of  metrics. 
The  conceptual  limitation  of  the  cost-based  approaches  to  generating  a  return  on 
investment-type  performance  ratio  is  that  they  do  not  have  a  surrogate  for  revenue 
(Johnson  1992).  The  problem  of  using  this  method  for  evaluating  the  value  added  by  IT, 
is  the  fact  that  if  cost  (or  any  of  its  derivatives)  is  used  as  a  surrogate  for  value,  then  all 
the  information  is  contained  in  one  term  of  the  ratio,  i.e.,  the  denominator  (Housel  et  ah, 
2001,  p.  10).  The  data  source  for  value  should  come  from  the  revenue  side  of  the  firm's 
performance  (i.e.,  numerator)  and  the  data  source  for  cost  (i.e.,  cost)  should  come  from 
the  cost  to  produce  the  firm's  outputs.  In  the  case  of  lEEM,  a  form  of  ABC  provides  a 
numerator  of  procedures  to  accomplish  an  activity  over  the  time  and  cost  it  takes.  This  is 
akin  to  and  borrows  from  the  KVA  methodology  and  presents  a  method  to  measure  and 
trace  value  at  the  sub-corporate  level,  ironically  unlike  ABC’s  originally  intended  design 
(see  Section  II.C.2:  Cost-Based  Similarities  and  Differences  and  Table  1,  Common 
Approaches  to  Measuring  the  Return  on  IT).  Nevertheless,  conventional  application  of 
ABC  is  strictly  about  cost.  This  approach  to  ROI  can  compliment  that  of  KVA  and  an 
illustration  of  this  follows. 

ABC  is  a  systematic,  cause  &  effect  method  of  assigning  the  cost  of 
activities  to  products,  services  and  customers  {cost  objects).  ABC  uses  a  simple 
principle: 

•  Products,  Services  and  Customers  generate  the  need  for  activities. 

•  Activities  consume  resources. 

•  The  more  varied  and  diverse  the  Products,  Services  and  Customers,  the  more 
activities  are  generated  and  the  more  resources  are  needed. 
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ABC  measures  the  eost  and  eonsumption  of  aetivities  and  assigns  these  eosts  only  to  the 
eost  objeet  generating  the  aetivity,  sueh  as  the  serviee  provided  or  the  demand  of  a 
eustomer  (Roztoeki,  2001,  p.  2).  ABC  introduees  the  eoneept  of  cost  drivers,  whieh  are 
any  faetor  that  eause  a  ehange  in  the  level  of  aetivity.  In  the  ease  of  lEEM  metries 
analysis  and  notions,  it  is  the  prooess  ohanges,  speoifioally  the  prooedures  removed, 
modified  or  introduoed.  It  is  the  ohoioe  and  use  of  eost  drivers  that  enables  the  analysis 
team  to  aoourately  allooate  the  indireot  and  overhead  eosts  to  the  appropriate  eost  objeet. 
Eor  example,  assigning  resouroe  eosts  assooiated  with  looking  and  prooessing 
information  to  do  a  job  (aetivity)  to  provide  a  serviee  (eost  objeet)  oan  be  aooomplished 
by  using  the  number  of  searohes,  navigational  steps,  oliokstreams,  and  other  metrics 
outlined  by  the  lEEM  metrics  baseline  and  conversion  ratios  (cost  driver).  The  better  the 
service  (be  it  faster  or  a  qualifiable  improvement),  the  less  time  (resource)  is  used  to  run 
through  the  steps  necessary  to  provide  the  service,  the  less  costs  are  assigned  to  this 
service.  By  focusing  on  the  minimization  or  optimization  of  an  activity  by  either 
reducing  the  number  of  procedures  or  strengthening  them  to  accomplish  an  activity 
through  crucial  process  changes  of  intranet  practices,  the  number  of  procedures  reduced 
provides  a  means  to  measure  consumption  of  resources  as  well  as  a  trace  to  where  value 
is  gained.  In  addition,  the  trace  on  value  further  informs  future  decisions  regarding  where 
process  changes  have  the  greatest  impact. 

Expenses  which  can  be  associated  with  a  particular  cost  object  are 
considered  “direcf’,  i.e.,  salaries  and  expenses  and  those  which  can  not  be  associated 
with  a  particular  cost  object  are  defined  as  “overhead”,  i.e.,  operational  costs.  It  is  these 
costs  that  can  be  traced  from  activities  to  cost  objects.  To  systematically  relate  activities 
to  cost  objects,  the  direct  and  overhead  costs  of  each  cost  object  are  added  together  as 
“indirect”  costs  in  order  to  obtain  the  product  cost.  The  product  cost  represents  an 
estimate  of  the  actual  expenditure  on  the  part  of  a  company  to  generate  a  cost  object, 
rather  than  the  cost  of  that  object  to  a  customer.  The  remaining  paragraphs  in  this  section 
provide  an  example  of  how  this  would  work  in  relation  to  lEEM. 
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An  example  of  ABC  derived  ROIMI  for  lEEM  ean  be  illustrated  by  a 
division  that  runs  two  sets  of  proeedures:  one  in  a  non-optimized  portal  and  the  other  in 
an  optimized  portal  essentially  using  a  similar  proeess  but  with  less  required  proeedures. 
Stepping  through  any  proeedures  to  do  nearly  any  aetivity  takes  time  and  resouree. 
Additionally,  proeess  ehanges  that  result  from  aetions  taken  from  metries  analysis  relate 
direetly  to  proeedures  taken  and  take  up  time  and  resourees  as  well  whieh  need  to  be 
taken  into  aeeount  when  seeking  a  ROE 

Eor  instanee,  during  a  30  day  period  to  aeeomplish  an  aetivity,  the  direet 
and  overhead  eost  assigned  to  a  seetion  is  $2,400.  An  optimized  portal’s  eost  would  need 
to  aeeount  for  the  eosts  devoted  in  optimizing  the  portal  for  that  given  period.  Eor 
instanee,  the  eosts  eould  amount  to  $800  and  this  eost  eould  be  aeeounted  for  during  the 
period  of  just  one  aetivity  or  amortized  over  a  longer  period.  It  would  be  more  realistie 
however  to  amortize  these  eosts  over  the  eourse  of  at  least  one  year  over  the  same 
aetivities  that  take  advantage  of  the  same  proeedure  ehanges  made  to  faeilitate  the 
eompletion  of  an  aetivity  as  a  part  of  doing  business,  i.e.,  “time  to  value”. 

This  aetivity  example  would  inelude  the  following: 

•  310  proeedures  are  required  to  perform  an  aetivity  in  a  portal  not  optimized 
(links,  design,  help). 

•  285  proeedures  are  required  to  perform  the  same  aetivity  in  an  optimized 

portal. 

•  Proeedures  in  a  non-optimized  portal  require  125  hours  of  work  to  be 
eompleted. 

•  Proeedures  in  an  optimized  portal  require  110  hours  of  work  to  be  eompleted. 

•  The  additional  eost  assoeiated  to  the  optimized  portal  for  this  aetivity  is  $800. 
In  total,  there  are  310  proeedures  to  eomplete  an  aetivity  in  125  hours  that  would 
normally  eost  $2,400  in  one  non-optimized  portal  versus  285  in  110  hours  for  an 
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optimized  portal  which  requires  an  addition  $800  to  cover  the  optimization  costs  (albeit 
for  one  activity  or  amortized  over  the  course  of  a  year); 


To  develop  this  example  further,  the  additional  costs  imposed  by  factoring 
in  the  costs  of  metrics  analysis  and  changes  are  factored  two  ways  in  the  two 
complimentary  tables  below  for  comparative  purposes:  in  one  30  day  activity  and  then 
amortized  over  the  course  of  a  year  for  the  same  activity  conducted  multiple  times  by  one 
knowledge  worker. 


Indirect  cost  (time)  assigned  procedures  in  non-optimized  portal  - 
(310/125  hours) 

1  procedure  per 
every  24  min.  12  sec 

Indirect  cost  (time)  assigned  procedures  in  optimized  portal  - 
(285  /no  hours) 

1  procedure  per 
every  23  min.  10  sec 

Average  indirect  cost  assigned  non-optimized  portal  over  time 
period  to  complete  the  activity  (285  /  $2,400) 

1  procedure  per 
every  $7.74 

Average  indirect  cost  assigned  optimized  portal  over  time  period  to 
complete  activity  (285/  $2,400  x  1  lOhours  /  310  hours  +  $800) 

1  procedure  per 
every  $10.21 

Average  indirect  cost  assigned  optimized  portal  amortized  over  one 
year  time  period  to  complete  the  activity 
(285/  $2,400  X  1  lOhours  /  125  hours  +  $800/12) 

1  procedure  per 
every  $7.64 

Table  6,  ABC  Costs  Assigned  to  Portals 


94 


Variable  Calculated 

Best  Case 

Worst  Case 

Difference 

Number  of  Procedures 

No.  procedures  required  to  perform  activity  in  a  non-optimized  portal 

310 

310 

No.  procedures  required  to  perform  activity  in  an  optimized  portal 

285 

300 

-15 

Number  of  Hours 

No.  hours  required  to  complete  activity  in  a  non-optimized  portal 

125 

125 

No.  hours  required  to  complete  activity  in  an  optimized  portal 

no 

116.5 

-6.5 

Number  of  Kuowledge  Workers  that  Couduct  same  Task 

500 

450 

-50 

Cost  of  Activity  (direct  and  Overhead) 

2400 

2400 

Additional  Cost  (Associated  to  the  optimized  portal  for  this  activity) 

800 

1100 

-300 

Indirect  Costs  for  Non-Optimized  Portal 

Indirect  cost  (time  in  minutes)  assigned  procedures  in  non-optimized  portal 

24.19354839 

24.19354839 

Average  cost  assigned  non-optimized  portal  over  period  to  complete  activity 

7.741935484 

7.741935484 

Indirect  Costs  for  Optimized  Portal 

Indirect  cost  (time  in  minutes)  assigned  procedures  in  optimized  portal 

23.15789474 

23.3 

-8.526315789 

Average  cost  assigned  optimized  portal  over  time  to  complete  activity 

10.21754386 

11.12266667 

-0.905122807 

Amortized  Indirect  Costs  for  Optimized  Portal 

Average  cost  assigned  optimized  portal  amortized  over  one  year  period 

7.644444444 

7.761555556 

0.117111111 

Frequency  of  Activity  (answer  only  one  choice  below) 

12 

12 

Weekly  (enter  the  value  of  1  if  this  applies) 

Bi-Weekly  (enter  the  value  of  1  if  this  applies) 

Monthly  (enter  the  value  of  1  if  this  applies) 

1 

1 

Bi-Monthly  (enter  the  value  of  1  if  this  applies) 

Quarterly  (enter  the  value  of  1  if  this  applies) 

Semi-Annually  (enter  the  value  of  1  if  this  applies) 

Annually  (enter  the  value  of  1  if  this  applies) 

Procedures  Gained  in  One  Activity  Period 

38.86363636 

21.88841202 

-16.97522435 

Percentage  Procedures  Gained  in  One  Activity  Period 

4.472140762 

3.834971618 

-0.637169144 

Procedures  Gained  in  One  Year  for  this  Activity 

166.3636364 

142.6609442 

-23.70269216 

Percentage  Change  in  Procedure  Productivity  for  Activity  in  1  Year  (%) 

4.472140762 

3.834971618 

-0.637169144 

Total  Cost  of  Procedures  for  this  Activity  in  One  Year 

29709.09091 

29980.25751 

271.1666016 

Savings  of  Procedures  for  Activity  in  Optimized  Portal  over  One  Year 

378.8856305 

-75.78568462 

-378.8856305 

Percentage  Savings  of  Procedures  in  Optimized  Portal  over  One  Year 

1.280019022 

-0.253463828 

-1.53348285 

Annual  Savings  to  Enterprise  for  Activity  (if  more  than  one  employee) 

189442.82 

-34103.5581 

-223546.3733 

Percentaae  Annual  Savings  to  Enterarise  for  Activife 

1.315502 

-0.26312241 

-1.578624428 

Additional  Similar  Activities  Accomplished  across  Enterprise 

291.8660287 

213.9914163 

-77.8746124 

Percentage  Additional  Similar  Activities  Accomplished  across  Enterprise 

4.864433812 

3.962804006 

-0.901629806 

Virtual  K-Workers  Gained  for  Activity  across  Enterprise 

24.32216906 

17.83261803 

-6.489551033 

Percentage  Virtual  K-Workers  Gained  for  Activity  across  Enterprise 

4.864433812 

3.962804006 

-0.901629806 

Table  7,  ABC  ROI  Returns  for  30  Day  Activities  During  One  Year 
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If  knowledge  workers  in  the  optimized  portal  worked  the  same  number  of 
hours  as  the  non-optimized  portal,  they  would  be  able  aceomplish  approximately  39  more 
proeedures  within  the  same  time  period  (310  hours  /  612  seconds  per  procedure).  Taken 
a  step  further  if  this  activity  is  done  12  time  in  the  course  of  a  year  (125  hours  is 
approximately  half  the  number  of  hours  one  person  works  a  month,  therefore  this  activity 
would  only  account  for  half  of  their  jobs)  and  the  costs  are  tabulated  using  the 
amortization  of  the  $800  development  costs  over  that  period,  the  total  number  of 
additional  procedures  accomplished  would  be  166  procedures  ([38.86  procedures 
accomplished  in  optimized  portal/  month  x  12  months]  -  [24.19  procedures  accomplished 
in  non-optimized  portal/  month  x  12  months  )  and  the  cost  would  be  $29,709.09  (3886.36 
procedures  /  year  x  12  months  x  $7.64  /  1  procedure). 

In  summary,  the  additional  166  procedures  gained  would  cost  an 
additional  $378.89  as  opposed  to  $909.09*,  providing  a  4.47%  increase  in  procedures 
accomplished  during  that  month.  Taken  collectively  across  the  enterprise  the  percentage 
increase  (or  ROI)  would  continue  to  gradually  climb  (albeit  slightly  in  some  activities 
and  indistinguishable  in  others  if  the  process  is  near  fully  optimized),  moreover  the  cost 
savings  would  be  profound.  The  example  given  is  prepared  in  the  context  of  what  one 
knowledge  worker  can  accomplish  in  the  average  amount  of  work  hours  per  month.  If 
the  savings  for  this  one  knowledge  worker  is  nearly  $379.89,  the  savings  would  be  even 
larger  every  year  when  applied  to  an  even  greater  number  of  them,  i.e.,  $379. 89x  500 
workers  who  must  accomplish  this  same  activity  equates  to  $189,442.82  annual  savings 
for  this  one  activity  alone.  Similarly,  an  additional  291  activities  can  be  accomplished 
(166  procedures  /  year  x  500  k-workers  =  83,181  procedures  /  285  procedures  per 
activity  accomplished  in  an  optimized  portal):  the  equivalent  of  adding  24  knowledge 
workers  (291  activities  /  12  activities  per  k-worker  per  year)  or,  alternatively,  providing  a 
justification  to  reduce  the  size  of  a  knowledge  worker  pool  who  completes  this  activity. 


(Total  Cost  of  all  procedures  completed  in  optimized  portal)  versus  (Total  Cost  of  procedures  in  non- 
optimized  portal  had  it  had  to  produce  the  same  number  of  procedures  as  the  optimized  portal  in  the  same 
period  of  time)  -($800  Total  Cost  of  activity  x  12  months) 
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Although  the  percentage  improvement  appears  nearly  insignificant  for  the 
first  activities,  the  impact  of  being  able  to  perform  more  procedures  with  the  same  cost  or 
less  procedures  to  accomplish  an  activity  in  less  time  is  significant  when  applied  across 
the  enterprise  for  all  knowledge  workers  (or  groups)  who  perform  this  same  activity  over 
time,  as  Figure  Productivity  Pyramid  below  illustrates  in  purple  (color  for  efficacy  gains) 
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This  type  of  calculating  is  conceivable  and  doable  for  standardized 
processes  of  work  that  can  be  enhanced  by  portal  changes  in  information  access  and 
discovery.  However,  it  is  limited  in  accounting  for  creative  processes  since  the  steps  of 
each  procedure  cannot  be  known  with  certainty  ahead  of  time.  It  can  provide 
approximate  accountability  by  gauging  levels  of  disintermediation  achieved  through 
conversion  rates,  anecdotal  confirmations  by  the  knowledge  workers  through  surveys  and 
more  general  time  estimates  derived  from  shorter  development  times  achieved  by  in- 
house  research  and  development  efforts  or  cycles  within  the  R&D  department  itself. 

The  example  of  procedures  used  above  can  take  on  other  parallel  meanings 
in  terms  of  value  that  can  be  derived.  For  example,  if  computer  code,  i.e.,  instruction  sets 
(another  surrogate  for  procedures),  is  discovered  as  a  result  of  it  being  more  accessible, 
there  would  also  be  significant  savings.  Using  the  information  provided  by  ABC  as  a 
means  to  monitor  impact  of  process  changes  (as  a  result  of  lEEM  metrics  analysis), 
companies  in  effect  can  cut  costs,  identify  opportunities  for  improvement,  and  determine 
a  more  profitable  way  of  conducting  business  activities.  In  addition,  the  output  of  the 
ABC  analysis  is  a  good  basis  for  revising  tactical-level  portal  changes/enhancements  as 
well  as  efficiencies  expected  of  corporate  portal  strategies  (see  Appendix  E:  Spreadsheet 
of  ABC  ROI  Return  Examples  to  see  a  variety  of  different  activities  conducted  at  varying 
intervals  each  year,  ie.,  weekly,  bi-weekly,  quarterly,  etc.).  Appendix  E  also  provides 
worst  and  best  case  estimates  of  process  changes  to  provide  BDMs  a  measure  of  risk 
involved  with  each  series  of  process  changes.  Eor  example,  in  the  Quarterly  Activity 
estimates,  the  best  case  is  an  annual  savings  gain  of  $428,571  but  poses  a  risk  of  -$35,200 
in  the  worst  case.  Given  this  range,  the  BDMs  can  better  ascertain  if  the  changes  are 
worth  the  risk. 
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ONE  YEAR  RESULTS 

Variable  Calculated 

Number  of  Procedures 

No.  procedures  required  to  perform  activity  in  a  non-optimized  portal 
No.  procedures  required  to  perform  activity  in  an  optimized  portal 

Number  of  Hours 

No.  hours  required  to  complete  activity  in  a  non-optimized  portal 
No.  hours  required  to  complete  activity  in  an  optimized  portal 

Number  of  Knowledge  Workers  that  Conduct  same  Task 
Cost  of  Activity  (direct  and  Overhead) 

Additional  Cost  (Associated  to  the  optimized  portal  for  this  activity) 


Weekly  Weekly  Bi-Weekly  Bi-Weekly  Monthly  Monthly  Quarterly  Quarterly 
Best  Case  Worst  Case  Best  Case  Worst  Case  Best  Case  Worst  Case  Best  Case  Worst  Case 


50 

45 

30 

26 

4000 

700 

200 


50 

48 

30 

28.5 

3500 

700 

300 


no 

95 

68 

57 

1000 

1200 

400 


no 

102 

68 

64 

900 

1200 

700 


310 

285 

125 

no 

500 

2400 

800 


310 

300 

125 

116.5 

450 

2400 

1100 


1200 

1100 

400 

350 

250 

15000 

1000 


1200 

1150 

400 

375 

220 

15000 

1400 


Frequency  of  Activity  (answer  only  1  choice) 


48 


48 


24 


24 


12 


12 


Procedures  Gained  in  One  Activity  Period  6.923  2.526  18.33  6.375  38.863  21.888 

Percentage  Procedures  Gained  in  One  Activity  Period  3.846  1.052  3.03  -1.477  4.472  3.834 

Procedures  Gained  in  One  Year  for  this  Activity  92.30  25.26  80  -39  166.363  142.66 

Percentage  Change  in  Procedure  Productivity  for  Activity  in  One  Year  3.846  1.052  3.03  -1.477  4.472  3.834 

Total  Cost  of  Procedures  for  this  Activity  in  One  Year_ 33830.76  33915.78  29277.19  29543.75  29709.09  29980.25 


157.142  76.666 

4.761  2.222 

228.57  106.667 

4.762  2.222 

61142.85  61493.333 


Savings  of  Procedures  for  Activity  in  Optimized  Portal  over  One  Year  1061 .538 

Percentage  Savings  of  Procedures  in  Optimized  Portal  over  One  Year  3.140 

4246153 

Annual  Savings  to  Enterprise  for  Activity  (if  more  than  one  employee) 

Percentage  Annual  Savings  to  Enterprise  for  Activity  3.159 

Additional  Similar  Activities  Accomplished  across  Enterprise  8205.128 

Percentage  Additional  Similar  Activities  Accomplished  across  Enterprise  4.273 

Virtual  K-Workers  Gained  for  Activity  across  Enterprise  170.94 

Percentage  Virtual  K-Workers  Gained  for  Activity  across  Enterprise  4.273 

TWO  YEAR  RESULTS 

Frequency  of  Activity  (answer  only  one  choice  below)  96 


37.89 

0.111 

132631.57 


0.112 

1842.105 

1.096 

38.37 

1.096 

96 


395.53 

1.354 


-1169.204 

-3.963 


378.885 

1.28 


-75.785 

-0.253 


1714.285 

2.81 


842.105 

3.508 

35.087 

3.5087 

48 


-344.117 

-1.593 

-14.338 

-1.593 

48 


291.866 

4.864 

24.322 

4.864 

24 


213.991 

3.9628 

17.832 

3.962 

24 


51.948 

5.1948 

12.987 

5.1948 


-160 

-0.26058 


20.4057 

2.31884 

5.101449 

2.31884 


Procedures  Gained  in  One  Activity  Period  6.923  2.526  18.33  6.375  38.863  21.888 

Percentage  Procedures  Gained  in  One  Activity  Period  3.846  1.052  3.03  -1.477  4.472  3.834 

Procedures  Gained  in  One  Year  for  this  Activity  184.61  50.52  160  -78  332.727  285.3218 

Percentage  Change  in  Procedure  Productivity  for  Activity  in  One  Year  3.846  1.052  3.03  -1.477  4.472  3.834 

Total  Cost  of  Procedures  for  this  Activity  in  One  Year_ 67430.76  67515.78  58077.19  58343.75  58509.09  58780.25 


157.142  76.667 

4.761  2.222 

457.142  213.333 

4.761  2.2222 

121142.8  121493.33 


Savings  of  Procedures  for  Activity  in  Optimized  Portal  over  One  Year 
Percentage  Savings  of  Procedures  in  Optimized  Portal  over  One  Year 

Annual  Savings  to  Enterprise  for  Activity  (if  more  than  one  employee) 

Percent^  Annual  Sayu^  to  En|^|>rise  for  A  _ 

Additional  Similar  Activities  Accomplished  across  Enterprise 
Percentage  Additional  Similar  Activities  Accomplished  across  Enterprise 
Virtual  K-Workers  Gained  for  Activity  across  Enterprise 
Percentage  Virtual  K-Workers  Gained  for  Activity  across  Enterprise 


2353.846  391.5789 


1268.261 


1666.86 


3.492  0.58  2.186 

9415384.62  1376526.M  1268261.56 


3.502 


1028.68  4571.428 

1.752 


16410.25 

4.273 

170.94 

4.273 


Table  8.  One  Year  vs.  Two  Year  ROI  Returns  for  Different  Activities 
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Despite  the  promising  benefits  of  this  technique  based  on  the  lEEM 
baseline  of  metrics  and  conversion  rates,  it  does  require  time  to  gauge  and  calculate, 
which  runs  counter-grain  to  the  fast  pace  nature  of  internet  economies.  An  illustration  of 
this  using  the  previous  30  day  example  is  depicted  in  Tables  8  {One  Year  versus  Two 
Year  ROI  Returns  for  Different  Activities)  which  illustrates  how  changes  in  worst  case 
estimates  can  go  from  a  negative  projection  (-$34,103.55  or  -0.263%  ROI)  after  one  year 
to  a  positive  return  after  two  years  ($462,908.76  or  1.785%  ROI).  On  the  contrary,  some 
investments  may  never  -  or  for  an  unacceptably  long  period  of  time  -  provide  a  positive 
return  under  worst  case  estimates,  such  as  the  bi-weekly  scenario  in  Table  8  during  one 
and  two  year  returns,  providing  negative  4.059%  and  2.768%  ROIs  respectively.  It  is  up 
to  the  BDMs  to  decide  what  is  an  acceptable  risk,  but  they  must  first  be  given  the 
expectations  in  terms  they  understand  and  which  can  be  rationalized  by  a  sound  model 
supported  by  mathematics  (see  Appendix  E:  Spreadsheet  of  ROI  Returns  for  more 
examples  and  an  explanation  of  the  calculations).  Given  the  time  it  would  take  to  realize 
value  gains  is  affordable,  ABC-like  estimates  of  savings  from  lEEM  metrics,  i.e.,  semi¬ 
annual  to  annual  results,  appear  to  be  a  good  managerial  tool  to  gauge  time  to  value  for 
intranets  (as  well  as  internets)  of  large  companies  involved  in  e-business. 


3.  Keys  to  Measuring  Returns  on  IT 

Although  they  are  inter-related,  time  is  the  efficiency  factor  and  creating  value  is 
the  effectiveness  factor.  A  tractable  method  to  prove  this  with  any  hard  numbers  would 
be  similar  to  the  examples  above  in  the  form  of  time  reductions  in  the  exercise  of 
procedural  knowledge  —  much  like  KVA  does  in  the  form  of  return  on  knowledge  in  case 
example  it  uses  showing  differences  of  learning  times  before  and  after  the  application  of 
knowledge  (Housel  et  al.  1999).  Consequently,  KVA  is  proposed  in  this  paper  as  one 
way  to  estimate  the  value-revenue  allocatable  to  corporate  assets  such  as  people  and 
technology.  And  ABC  is  proposed  as  another  means  to  estimating  the  return  or  cost- 
benefit  of  ROIMI  in  a  tractable  procedure  presented  in  terms  of  value  gain,  be  it  cost, 
time,  activities  generated  or  number  of  knowledge  workers  required.  In  both 
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methodologies,  ereating  value  ean  be  eonveyed  through  the  inerease  of  eonversion  ratios 
(found  in  the  metries  sub-grouping  of  Appendix  D)  that  constitute  critical  business 
requirements,  such  as  loyalty,  reach  and  disintermediation. 

KVA  and  ABC  methods  of  estimating  ROI  also  compliment  each  other  when 
used  together  to  estimate  the  same  process.  For  example,  audits  that  result  due  to 
discrepancies  can  be  automated  quickly  while  others  are  more  manual  intensive  and 
require  time  to  resolve.  In  any  event,  significant  cost  savings  can  be  made  if  the  number 
of  discrepancies  that  require  audits  is  lowered.  In  one  study,  KVA  analysis  does  not 
make  any  recommendations  for  changes  in  auditing  function  because  on  paper  it  has 
relatively  high  ROK  (cost  to  learn  how  to  conduct  audits  divided  into  the  revenue  created 
by  them  when  factoring  associated  percentage  of  costs  of  audits  and  the  revenue 
generated  back).  From  an  ABC  perspective,  however,  the  auditing  function  comes  under 
scrutiny  because  of  the  high  cost  when  a  discrepancy  has  to  be  researched.  Intuitively, 
auditing  does  not  add  value  for  the  customer  and,  therefore,  is  a  target  for  re-engineering 
discovered  by  ABC  (Nomura,  2002).  Thus,  when  using  KVA,  an  analyst  must  be  careful 
to  factor  in  qualitative  measures  (i.e.,  common  sense)  to  ensure  a  thorough  and  complete 
re-engineering  effort  is  made. 

On  the  other  hand,  ABC  has  deficiencies  when  dealing  with  processes  that  are 
complex  or  involve  a  large  amount  of  knowledge.  In  such  processes,  costs  and  ROK  will 
not  be  correlated  and,  consequently,  re-engineering  efforts  will  be  focused  in  different 
areas.  As  a  result,  ABC  may  misdirect  re-engineering  efforts.  As  the  economy  shifts 
from  a  manufacturing  to  a  services  emphasis,  the  value  of  the  KVA  methodology 
increases.  Knowledge  intensive  processes  are  more  prevalent  in  the  services  sector  and, 
therefore,  will  benefit  the  most  from  a  re-engineering  project  using  the  KVA 
methodology.  Thus,  while  ABC  is  useful,  due  to  our  ascent  into  the  "Information  Age", 
KVA  appears  to  be  more  relevant  for  the  future  (Nomura,  2002). 
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The  advantage  of  eombining  lEEM  and  its  assoeiated  baseline  of  metries  whieh 
emphasizes  surveys  and  ineludes  the  key  eonversion  ratios  to  estimate  the  improvement 
of  eritieal  business  requirements  with  ROI  estimates  from  both  KVA  and  ABC  is  that 
they  eolleetively  overeome  a  limitation  of  the  KVA  and  ABC  techniques  when  applied 
alone,  not  readily  representing  to  the  same  extent  increases  in  quality,  creativity  and 
knowledge  worker  satisfaction.  However,  these  factors  do  impact  the  bottom  line  and 
will  eventually  find  their  way  into  processes  with  predetermined  outputs  because  the 
most  intangible  asset  of  employee  knowledge  eventually  becomes  a  tangible  asset 
embedded  in  company  IT.  Though  it  is  unlikely  that  the  benefits  of  these  factors  will 
ever  be  completely  quantified,  over  time  this  approach  does  allow  for  eventual 
accountability  of  conversions  such  as  creative  outputs  into  value  since  they  are  inevitably 
embedded  into  processes  with  predetermined  outputs  (Housel  et  al,  2001).  Taken  as  a 
whole,  conclusions  from  these  approaches  should  be  plausible  and  provide  management 
with  a  more  comprehensive  picture  of  the  value  and  direction  of  their  intranet 
refinements  and  initiatives  than  they  currently  get  from  any  other  means  or  methodology. 

As  the  academic  community  points  out,  there  are  four  key  issues  that  need  to  be 
addressed  within  any  framework  for  measuring  the  return  on  IT. 

•  Unambiguous  allocation  of  value  as  well  as  cost  of  IT  initiatives 

•  Mapping  of  IT  economic  impacts  at  any  level  of  aggregation 

•  Common  unit  of  measurement 

•  A  supporting  theoretical  framework 

Together  the  lEEM  and  its  supporting  metric  conversion  ratios  and  their  analysis  along 
with  the  application  of  a  form  of  KVA  and  ABC  to  determine  ROI  in  measurable 
common  units  meet  all  four  of  these  essential  requirements.  The  collective  framework 
and  approach  is  theoretically-based  and  “operationalizable”.  Eurther,  such  a  framework 
can  prove  useful  to  the  practitioners  who  are  struggling  to  determine  which  IT  process 
designs  will  provide  the  best  returns  from  their  intranet  portals.  In  effect,  consistent 
application  of  the  lEEM  framework  and  baseline  metrics  in  combination  with  KVA  and 
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ABC  improves  insight  into  how  to  increase  the  value  of  an  enterprise  in  a  rational  fashion 
using  common  units  of  measurement  when  and  where  necessary. 


G.  SUMMARY 

ROI  is  one  of  those  things  that,  in  theory,  makes  perfect  sense.  The  problem  with 
relying  solely  upon  financial  techniques  such  as  Net  Present  Value  is  that  they  don't 
necessarily  capture  all  of  the  business  benefits  of  an  IT  investment,  nor  do  they  help  to 
evaluate  all  of  the  options  available  (see  Appendix  E,  Common  Approaches  to  IT  ROI). 
Nonetheless,  a  rational  and  comprehensive  pursuit  of  ROI  can  lead  to  the  discovery  and 
optimization  of  proper  metrics  that  can  both  demonstrate  the  business  value  of  intranets 
portals  as  well  as  guide  efforts  toward  enhancements  to  them  that  will  have  the  greatest 
ROI. 


Although  costs  vary  widely,  executives  and  knowledge  workers  down  the  line 
uniformly  expect  big  benefits  from  intranet  portals.  Ironically,  however,  firms  will  spend 
big  money  but  won’t  measure  the  results.  Sixty-one  percent  of  firms  don’t  have  any 
metrics  to  prove  portal  benefits  (Gillet,  2001,  p.  7).  Reasons  for  the  lack  of  measurement 
range  from  not  knowing  how  to  do  this  to  taking  the  easiest,  albeit  not  very  insightful, 
course  of  action  to  collect  simple  metrics  such  as  page  hits.  Assessing  soft  and  derived 
benefits  for  intranets  —  such  as  improved  customer  service,  satisfaction,  collaboration, 
loyalty  and  quicker  time  to  market  —  can  be  one  of  the  most  challenging  tasks  in 
determining  ROI  for  intranet  portals.  Thus,  by  applying  the  lEEM  metrics  baseline, 
conversion  ratios  and  analysis,  an  azimuth  indicator  showing  how  well  a  corporation  is 
reaching  and  supporting  its  strategic  business  requirements  is  possible,  provided  a 
reasonable  return  on  intranet  metrics  investment  (ROIMI)  of  the  costs  of  the  analysis 
process  is  compared  to  the  time  benefits  using  a  form  of  KVA  and  ABC.  Exactly  who 
should  be  designated  to  ensure  metric  results  and  analysis  produce  reasonable  estimates 
of  ROIMI  is  addressed  next  in  Chapter  VI,  Business  Intelligence  Team. 
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VI.  BUSINESS  INTELLIGENCE  TEAM 


IT,  in  particular,  needs  to  be  managed  as  a  capital  investment 

opportunity,  rather  than  being  merely  a  money  pit  of  expense. 

John  Berry,  IT  Consultant  and  Columnist 

A,  ROLE  OF  BUSINESS  INTELLIGENCE  TEAM 

Successful  IT  projects  and  ROI  require  eollaboration  and  commitment  between 
business  managers  and  IT  professionals.  Because  analytie  skills  are  extremely  searee, 
enterprises  eannot  afford  to  seatter  these  experts.  Therefore,  enterprises  should  ereate  a 
Business  Intelligenee  Team  (BIT),  in  whieh  analysts  and  IT  experts  work  elosely  together 
to  support  business  managers  in  their  deeision-making  tasks.  The  BIT  should  be  housed 
where  it  matters  most  -  not  too  high  diseonneeted  from  the  real  world  and  not  too  low 
losing  an  over-arehing  view. 

Creation  of  a  BIT  is  important  beeause  IT  professionals  are  increasingly 
involuntarily  and  voluntarily  involved  in  the  strategie  and  financial  implications  of  IT. 
Involuntarily,  more  and  more  proposals  for  IT  projeets  must  inelude  eeonomie  and 
strategie  justifications  and,  similarly,  more  and  more  software  developers/developments 
are  being  pressured  by  eustomers  and  managers  alike  to  justify  the  value  of  the  software 
and  IT  enhancements  being  delivered.  Voluntarily,  developers  of  new  IT  produets  and 
proeesses  are  now  trying  to  promote  them  not  only  with  operational  justifieations  (faster, 
fewer  errors,  etc.)  but  with  strategic  and  financial  justifications  (“this  new  proeess  will 
deliver  more  business  value”).  Although  IT  professionals  are  inereasingly  involved,  two- 
thirds  of  eompanies  interviewed  by  one  research  group  stated  funding  responsibility  lay 
fully  with  IT  —  or  IT  had  no  role  at  all  (Gillet,  2001,  p.7).  This  doesn’t  mateh  up  with  the 
levels  and  types  of  responsibilities  and  expertise  required  to  deduet  and  ealeulate  IT 
related  ROI. 

B,  BIT  COMPOSITION 

Suecess  in  applying  Web  analyties  requires  eollaboration  between  analysts, 

portals  managers  and  BDMs.  Having  the  neeessary  skills  in  these  three  fields  to  collect, 

105 


interpret  and  act  on  information  quickly  is  a  competitive  e-business  differentiator. 
Preferably,  the  people  involved  in  Web  analytics  should  be  skilled  in  more  than  one 
discipline.  Web  analysis  is,  first  of  all,  an  analytics  exercise,  not  an  IT  project. 

1,  Analyst 

An  analyst  must  be  capable  of: 

•  Exploring  data  and  discovering  patterns,  meaningful  relationships,  and 
anomalies. 

•  Working  with  the  IT  department  to  develop  insight  into  how  to  identify  data 
from  the  Web  and  other  sources  for  a  specific  analysis. 

•  Using  a  palette  of  techniques,  ranging  from  simple  data  aggregation,  via 
statistical  analysis,  to  complex  data  mining. 

•  Being  fluent  with  analytics  tools. 

•  Distilling  the  relevant  parts  and  producing  sound  recommendations,  based  on 
the  right  set  of  metrics. 

2,  IT  Professional 

An  IT  professional  must  be  capable  of: 

•  Working  with  a  business  manager  to  define  the  right  requirements. 

•  Implementing  the  required  changes  in  an  efficient  way. 

•  Supporting  analysts  by  advising  on  the  efficient  storage  of  Web  site  data, 
integration  of  data  with  other  sources  and  how  to  keep  an  historic  overview  of  this  data. 

•  Working  with  analysts  to  identify  the  necessary  data  for  a  specific  analysis 
and  make  sure  it  is  accessible  and  understandable. 
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3,  Business  Manager 

A  business  manager,  responsible  for  a  portal  site,  must  be  capable  of: 

•  Interpreting  the  results  and  creating  decision  alternatives,  together  with  an 
analyst  and  IT  professional/portal  manager. 

•  Tying  analytic  results  to  the  corporate  objectives  to  ensure  that  decisions 
support  the  enterprise's  business  requirements. 

•  Initiating  process  changes  based  on  changes  in  portal  sites  and  to  follow  up  on 
triggers  or  anomalies  reported  from  the  respective  intranet  portals. 


Business 
Skills 


Establish 
requirement 

Monitor 

Link  to  corporate  \results 
strategy 


Figure  15,  Complimenting  Skill  Sets  and  Tasks  for  BIT  ^ 


C.  BIT  ADVANTAGES  AND  SHORTCOMINGS 

While  portal  managers  tend  to  have  a  short-range  view;  the  BDMs  have  a  long- 
range  perspective  and  while  portal  managers  ask  how  and  when;  the  BDMs  ask  what  and 


^  Buytendijk,  2001,  http://www.gl.iit.edu/gartner2/research/97600/97608/97608.html 
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why?  The  analyst  meanwhile  strives  to  figure  out  as  many  answers  to  these  questions  as 
possible.  Creation  of  a  multi-diseiplinary  and  perspeetive  BIT  produees  more  advantages 
than  shortfalls,  partieularly  for  large  intranets,  in  guiding  IT  projeets  and  determining 
their  ROT  Advantages  inelude: 

•  One  group  to  eentralize  a  variety  of  analytie  tasks  (see  seetion  IV. B. 4,  A 
Single  Reporting  Service). 

•  BDMs  are  involved  up  front  during  ROI  benefits  analysis  and  ean  be  held 
more  aeeountable  along  with  IT  for  aehieving  the  predieted  benefits. 

•  Engenders  business  aeumen  of  teohnieal  workers. 

•  Ineorporates  a  better  mix  of  eommon  sense,  professional  judgment, 
quantitative  modeling  and  strategie  perspeetive. 

•  More  likely  to  ferret  out  benefits  buried  in  other  lines  of  business  -  due,  in 
part,  to  their  eolleetive  multi-diseiplinary  baekgrounds. 

•  Separate  software  proposals  into  those  that  have  potential  for  ROI  and  those 
that  are  simply  the  eost  of  doing  business  (i.e.,  know  when  ROI  justifieation  is  warranted) 

•  Provides  greater  validity  to  ROI  analysis,  resulting  in  a  wider  spread  of 
aeeeptanee. 

Disadvantages  tend  to  affeet  smaller  organizations  and  inelude: 

•  The  Web  exaeerbates  the  advantages  of  seale:  A  large  enterprise  will  get  far 
more  leverage  from  its  Web  investments  than  a  small  one  (Casser,  2001,  p.3). 

•  Some  assets  used  to  eompute  value-based  measures  are  in  the  form  of 
intelleetual  eapital,  whieh  runs  up  against  a  faseinating  set  of  issues  in  figuring  out  how 
to  value  this  talent  (Meyers,  1997,  p.  47).  The  result  is  that  value-based  performanee 
metries  make  little  sense  for  eompanies  without  signifieant  hard  assets. 

•  Depending  on  their  levels  of  experienee,  some  BITs  may  make  the  mistake  of 
thinking  that  measurement  of  eausation  is  mueh  more  prevalent  than  it  really  is, 
partieularly  if  too  many  modifieations  are  made  to  portal  sites  at  onee  (see  Figure  8, 
Balancing  Changes  with  Cause  and  Effect). 
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•  When  BITs  are  replaeed  with  outside  vendors/analysts  to  measure  results, 
their  “outsider”  replacement  may  not  know  the  internal  culture  issues  or  the  nuances  of 
the  enterprise,  placing  it  at  a  disadvantage. 

•  If  consultants  are  hired,  they  charge  steep  fees  for  this  type  of  service:  six 
figures  for  larger  companies  or  between  $15,000  -  25,000  per  month  for  smaller  ones. 

Depending  on  the  amount  charged,  outsourcing  may  be  an  acceptable  solution  for 
some  companies  that  simply  do  not  have  the  money  and  resources  to  invest  in  these 
skills.  However  the  costs  associated  with  this  may  risk  achieving  positive  ROIMI 
returns,  particularly  for  smaller  companies.  In  addition,  many  organizations  are 
uncomfortable  with  the  service  provider  model  because  sensitive  customer  data  is 
handled  by  a  third  party.  Organizations  are  also  at  the  mercy  of  the  service  provider  for 
report  customization  and  data  retention  policies.  Generally,  more  money  is  saved  by 
creating  BITs  in-house  and  the  personal  investments  are  greater  and  more  reliable  as 
well.  Thus,  corporations  should  consider  seeking  out  its  employees  who  best  meet  the 
BIT-related  analytic  skills  needed  (see  Figure  14,  Complimenting  Skill  sets  and  Tasks  for 
BIT)  for  intranet  portal  analysis  during  the  intermediate  and  developed  terms  because  the 
right  mix  can  create  a  competitive  advantage. 

Teams  selected  to  analyze  Web  metrics  should  try  to  keep  choices  tactical  in 
support  of  strategic  objectives,  i.e.,  making  changes  in  which  the  impact  is  generally 
known  and  can  be  measured  tractably.  For  example,  they  must  evaluate  their  success  or 
failure  by  tracking  metrics  for  each  strategic  objective,  not  solely  metrics  for  each  tactical 
initiative.  By  proceeding  in  this  manner,  they  can  overcome  a  variety  of  challenges  (see 
below  Figures  13,  Top  Challenges  in  Selecting  and  e-Business  Project,  and  14,  Top 
Challenges  Measuring  Success  of  e-Business  Investment)  now  confronting  commercial 
and  government  sectors  in  trying  to  measure  their  intranet  portals: 
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What  are  your  top  challenges  in  selecting 
which  eBusiness  projects  to  fund  ? 


Integrating  eBusiness  into 
Overall  Strategy 


Insufficient  Resources 


Political  Resistance 


Prioritizing  According  to 
Merit,  not  Hype 


Lack  of  Objective  Data 


Percent  of  50  eBusiness  Executives  Interviewed 
(Multiple  responses  accepted) 


Figure  16,  Top  Challenges  in  Selecting  and  e-Business  Project 


What  are  your  top  challenges  for  measuring 
the  success  of  eBusiness  investments 


Attributing  Incremental  Value 
to  Specific  Initiatives 

Determining  Customer 
Impaet 

Lack  of  Objective  Metrics 


Lack  of  Time  to  Evaluate 

Limited  Technical  Resources 


Percent  of  45  eBusiness  Executives  Measuring  Success 
(Multiple  responses  accepted) 


Figure  17,  Top  Challenges  Measuring  Success  of  e-Business  Investment 


Cameron,  2000,  p.  4. 
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D,  DISCOUNTING 


BITs  can  also  help  separate  soft  benefits  from  the  hard  benefits  any  given  projeet 
or  enhaneement  is  expeeted  to  aehieve.  In  addition,  they  ean  diseount  soft  benefits  where 
appropriate.  These  diseounts  help  to  hedge  against  rosy  projeetions  and  can  be  tailored  to 
speeifie  groups.  For  example,  white-eollar  knowledge  workers  may  have  their 
produetivity  savings  from  software  diseounted  by  up  to  80  pereent,  while  faetory  workers 
and  salespeople  may  be  diseounted  by  20  pereent.  The  point  being  that  any  soft  faetor 
ean  be  given  a  numerieal  weight  to  aeeount  for  its  understood  and  agreed  upon  impaet 
with  regard  to  a  speeifie  line  of  work.  Thus,  while  targeting  high-end  knowledge  workers 
where  the  gains  really  matter  most  (see  seetion  V.D.2.d,  High-End  Knowledge  Workers), 
BITs  ean  exereise  diseretion  in  not  over  projeeting.  However,  if  they  “over-engineer”  the 
proeess,  people  may  walk  away  from  the  results.  The  predietors  and  weights  used  will  no 
doubt  beeome  more  aceurate  with  praetiee  gradually  over  time.  This  phenomenon  — 
whieh  is  unique  for  every  eompany  —  aeeounts  for  the  laek  of  weights  provided  in  the 
lEEM  set  of  baseline  metries. 

Along  these  lines,  there  are  a  variety  of  vendors  supplying  tools  that  measure 
metries  in  various  ways.  Unfortunately,  few  of  these  metries  are  indieators  of  business 
performanee  and,  taken  out  of  eontext,  these  "standard"  metries  ean  lead  to  poor 
eonelusions  or  no  eonelusions  at  all.  Henee,  enterprises  need  to  think  hard  about  how 
they  will  apply  Web  analyties  to  aehieve  tangible  business  results.  Creating  their  own 
Business  Intelligenee  Team  to  apply  a  set  of  baseline  metries  and  relevant  eonversion 
ratios  that  aeeount  for  eritieal  business  requirements,  like  those  outlined  within  the 
framework  of  lEEM,  is  one  method  to  start  their  measurement  proeess  in  a  standard, 
uniform  manner  through  a  single  reporting  serviee. 


"  Cameron,  2000,  p.  5. 
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E,  SUMMARY 

In  the  final  analysis,  Web  analytics  are  an  immature  discipline  that  requires  the 
support  and  intelligence  the  human  cognitive  factors  a  BIT  inherently  brings.  Within  the 
lEEM  framework,  BITs  can  develop  business  cases  that  will  help  senior  management 
better  understand  the  value  of  a  particular  IT  investment  as  it  supports  a  variety  of 
business  requirements.  Their  task  of  tying  metrics  back  to  business  justification  is 
essential  for  determining  success  of  intranets  portals  metrics  and  communicating  the  big 
picture.  Naturally,  the  approach  and  empowerment  (i.e.,  the  extent  to  which  they  can 
recommend  and  enforce  best  practices  across  the  enterprise  based  on  their  analysis)  a  BIT 
is  allowed  will  depend  in  part  on  the  style  of  the  manager,  the  culture  of  the  company, 
and  support  from  top  executives.  Calculating  and  communicating  their  intranet  analytic 
results  will: 

•  Enhance  the  standing  of  IT  in  eyes  of  business  management. 

•  Cut  through  culture  issues. 

•  Eead  to  better  choices  among  potential  IT  portal  initiatives. 

•  Eead  to  yet  better  alignment  of  IT  and  business  goals. 

•  Provide  feedback  so  IT  can  improve  itself  over  time  (hence,  competitiveness). 

•  Provide  a  forum  and  format  that  both  facilitates  and  requires  managers  to 
detail  how  process  changes  will  add  business  value. 

•  Posters  sponsorship  and  collaboration  among  departments  and  disciplines, 
preventing  blind  spots  holding  more  people  accountable. 

Along  with  the  application  of  the  metrics  baseline  and  conversion  rates  to  begin,  adjust 
and  continue  measuring  performance  success,  the  complimentary  skills  and 
empowerment  of  BIT  BDMs,  portal  managers  and  analysts  ensure  the  azimuth  indicator 
of  lEEM  remains  pointed  toward  value:  Internal  awareness  and  responsiveness  facilitates 
external  awareness  and  responsiveness. 

In  short,  it's  the  process,  and  not  technology,  that  makes  for  effective  IT 
governance.  The  BIT  works  together  to  better  orchestrate  the  impactful  processes  of 
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corporate  portals.  Before  it  gets  involved  in  any  tools,  the  BIT  ensures  processes  are  in 
place  and  understood  by  a  wider  audience  at  many  levels,  instilling  discipline  and 
acceptance.  These  proeesses  involve  inventorying  IT  resources,  ineluding  skills, 
hardware,  and  software;  analyzing  their  use  by  business  goal,  risk,  budget,  and  expected 
return;  and  seenario  planning.  The  eombination  of  process  and  technology  in  this  manner 
by  these  people  enables  all  knowledge  workers  to  be  more  productive. 
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VII.  CONCLUSION 


The  newest  innovations,  whieh  we  label  information  teehnologies, 
have  begun  to  alter  the  manner  in  which  we  do  business  and  create  value, 
often  in  ways  not  readily  foreseeable  even  five  years  ago. 

Alan  Greenspan,  Chairman,  US  Federal  Reserve 

A,  SUMMARY 

The  advent  of  intranet  portals  creates  new  opportunities  for  corporations  to  make 
capital  investments  in  their  own  knowledge  and  streamlining  of  working  processes  at  a 
rate  of  return  far  better  than  keeping  the  money  in  the  bank.  Proper  metrics  can  both 
demonstrate  the  business  value  of  intranets  and  portals  as  well  as  guide  efforts  toward 
enhancements  that  will  have  the  greatest  ROT  Only  firms  that  holistically  consider 
organizational  factors,  information  technology,  and  work  processes  will  be  able  to 
identify  such  solutions  to  act  upon.  Because  the  intranet  and  its  applications  suites  will 
have  a  measurable  impact  on  traditional  businesses,  more  companies  must  embrace  a  set 
of  metrics  that  gives  the  IT  initiative  in  this  area  credit  not  only  for  its  customers,  but  also 
its  overall  contribution  to  the  corporation  at  large. 

Organizations  should  predetermine  the  metrics  they  will  collect,  targeting 
customer  loyalty,  partner/affiliate  assessment,  content  effectiveness,  channel  efficiency, 
etc.  Web  analytics  tools  help  enterprises  jump-start  initiatives  and  provide  inspiration, 
but  the  results  are  worthless  without  context.  Along  these  lines,  enterprises  must  develop 
a  relevant  plan  of  attack  for  measuring  and  monitoring  their  website  data.  By  using  the 
Intranet  Efficiency  and  Effectiveness  Model  as  part  of  a  holistic  approach  toward  a 
comprehensive  appreciation  of  the  interplay  between  business  requirements  and 
underlying  intranet  segments,  constituents  and  relevant  metrics,  enterprises  will  gain 
valuable  insight  into  how  their  websites  are  performing  and  how  their  users  are 
interacting  with  their  offerings. 

Invariably,  a  numerical  model  must  be  created  and  the  most  difficult  part  of  this  is 

identifying  assumptions  regarding  costs  and  revenue  generation.  There  is  no  hard  and 
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fast  rule  for  defining  these  assumptions.  However,  a  eomprehensive  model  that  stresses 
strategie,  functional  and  technical  fit,  and  seeks  to  identify  opportunities  for  process 
improvements,  is  more  likely  to  result  in  top  management  support  because  it  is  based  on 
their  direction  for  the  organization,  not  merely  the  merits  of  a  technology  or  another  half- 
baked  technique  to  manipulate  numbers. 

The  goal  is  to  ensure  that  any  IT  investment  decision  can  be  shown  to  be 
consistent  with  the  organization’s  business  objectives.  It  may  take  some  experimentation, 
but  eventually  a  set  of  metrics  measurements  can  be  created  that,  collectively  and 
associated  to  the  lEEM,  describes  different  facets  of  how  well  a  corporation  is  achieving 
its  critical  business  requirements  via  its  intranet  portals  initiatives  large  and  small.  These 
measurements  should  be  codified  and  used  consistently  within  a  company;  this  will 
ensure  that  ROI  results  from  different  department  portals  are  comparable  to  each  other. 
Perhaps  the  most  important  benefit  of  applying  the  lEEM  and  metrics  is  that  the  business 
intelligence  team  will  learn  what  they  can  do  to  better  increase  value  through  leading 
periodic  re-examination  of  assumptions  and  results,  applying  experimentation,  rewarding 
collaboration  among  IT,  finance  and  business  units,  and  promoting  accountability.  After 
all,  a  group  can  only  improve  when  it  has  a  chance  to  lean  from  both  its  mistakes  and 
successes.  It  is  the  intent  of  the  lEEM  and  its  associated  metrics  analytics  to  be  just  such 
a  starting  place. 

B,  CONTRIBUTION  LIMITATIONS  AND  IMPLICATIONS 

This  research  provides  the  first  theoretical  model  for  the  Family  of  Measures 
approach  to  measuring  Web  activity  as  well  as  a  holistic  framework  and  multi¬ 
disciplinary  approach  not  previously  derived  in  viewing  and  measuring  intranet 
contributions  in  the  context  of  a  corporation’s  overall  strategic  business  requirements  by 
applying  a  balanced  baseline  set  of  metrics  and  conversion  ratios  (which  merge  website 
traffic  data  and  user  behavior)  linked  to  business  processes  as  they  relate  to  knowledge 
workers,  portal  managers  and  business  decision  makers.  It  also  outlines  how  this  should 
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be  done  and  by  whom  in  the  form  of  periodie  reviews  and  a  business  intelligenee  team 
respectively.  Lastly,  it  provides  a  means  -  as  well  as  a  justification  —  in  which  to 
calculate  ROI  for  the  metrics  investment  (time  saved  and  better  used  by  enhancements  to 
processes)  with  a  common  unit  of  analysis:  time,  which  can  always  be  translated  into 
money  saved  or  earned  and  competitive  advantage. 

However  there  are  limitations  that  need  to  be  taken  into  account  in  exercising  and 
improving  this  model  and  approach.  The  lEEM  metrics  and  BITs  are  intended  primarily 
for  large  corporations  who  have  multiple  portals  and  a  large  number  of  knowledge 
workers.  While  the  early  promise  of  the  Web  was  as  a  "great  equalizer",  the  reality  is 
that  the  Web  exacerbates  the  advantages  of  scale.  Although  corporations  with  smaller 
intranets  should  not  be  dissuaded  from  using  lEEM  metrics,  those  with  larger  intranets 
will  realize  greater  returns  on  the  investment  made  into  the  metrics.  At  some  point 
however,  applying  and  realizing  lEEM  metrics  benefits  does  not  make  financial  sense  if 
the  intranet  and  initiative  are  small. 

Another  limitation  to  this  approach  is  that  it  takes  time  in  order  to  measure 
changes  sequentially  and  incrementally.  In  addition,  if  too  many  changes  are  made  at 
once,  their  impact  may  become  too  diffuse  or  complex  to  estimate  and  trace  credibly  and 
accurately.  At  a  minimum,  it  would  take  about  75  days  before  any  comparisons  can  be 
made  from  two  30-day  sets  of  results  and  preferably  longer,  one  to  two  quarters  with 
several  30-day  result  sets  for  example,  before  reliable  and  stable  calculations/estimations 
are  filtered  and  appropriate  corrective  action  is  taken  (see  Section  IV.C:  Periodic 
Review).  The  larger  the  organization,  the  longer  the  analysis  will  take.  It  would  be  wise 
therefore  to  begin  small  and  incrementally  include  one  or  two  portals  at  a  time  until  all 
portals  that  need  to  be  measured  are.  This  gradual  expansion,  however,  will  require  a 
longer  period  of  time  to  canvas  the  corporation.  In  addition,  the  number  and  types  of 
changes  must  be  meticulously  recorded  and  compared  to  ensure  legitimate  undesirable 
patterns  are  recognized  and  known  remedies  are  effectively  and  immediately  applied. 


117 


There  may  also  be  competing  departmental  portal  interest  with  respect  to  the  time 
and  resources  the  BIT  can  apply  toward  taking  corrective  action.  Such  circumstances  can 
become  political  and  the  BIT  needs  to  be  prepared  to  list  and  justify  the  enhancements 
they  will  undertake  in  both  financial  and  effectiveness  priorities  (an  extension  of  their 
own  ROIMI  and  the  lEEM  metrics  prioritization  tables  in  Appendix  D,  Intranet  Portal 
Metric  Breakdown).  As  the  number  of  portals  and  portal  features  increase  the  number  of 
aggregate  changes  called  for  may  also  increase.  Another  surge  in  change  requirements 
may  be  by  necessity  due  to  the  advent  of  a  new  Web  technology.  Thus,  BITs  need  to  be 
empowered  to  make  best  practices  that  meet  the  dynamics  of  change  and  new  business 
needs  into  common  business  rules  and  policies  that  can  be  applied  and  enforced  in  short 
order  when  and  where  necessary  across  the  entire  enterprise.  Eack  of  this  power  is  a 
serious  limitation  and  will  hinder  progress  toward  achieving  greater  value. 

As  portals  mature  they  gradually  borrow  content  from  other  portals  and  data- 
warehouses  that  they  may  have  little  control  in  changing.  These  sister  content  providers 
may  be  tasked  to  farm  out  their  repositories  of  information  (for  example,  news,  research, 
archives  or  Human  Resource  related  data).  As  a  result,  some  portals  owners  may 
consider  the  lEEM  metrics  not  to  be  a  fair  estimate  of  their  value  creation  because  they 
are  getting  a  large  percentage  of  their  content  from  other  internal  sources  to  meet  the 
demands  of  their  knowledge  workers.  This  situation  is  not  necessarily  a  weakness  in  the 
lEEM  because  the  metrics  will  determine  which  portals  are  the  most  trafficked  and 
trusted  and  from  exactly  where  and  whom.  This  eventually  leads  to  resource  allocations 
and  is  why  some  portal  managers  fret  over  the  notion  that  their  significance  may  be 
diminished  by  a  popular  portal  that  does  little  else  than  provide  news  updates  (a  loose 
metaphor  being  the  impact  of  CNN  to  syndicate  channel  ratings).  Consequently,  all 
assumptions  and  calculations  along  with  concerns  about  comparisons  between  the 
participating  portals  need  to  be  addressed  in  a  reasonable  and  sensitive  manner  if  the 
analysis  results  are  to  be  perceived  as  valid  by  the  portal  players.  Keeping  all  portal 
managers  in  the  analytics  loop  is  crucial  therefore  and  should  be  scheduled  routinely. 
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C.  FUTURE  WORK 

Successful  Web  analyties  are  more  a  matter  of  skills  than  a  matter  of  technology. 
Nevertheless,  Web  logs  need  to  be  made  as  automated  and  quantitative  as  possible  whieh 
presents  a  number  of  ehallenges  to  deeision  makers  when  used  for  intranet 
measurements.  Understanding  eause-and-effeet  is  essential  to  the  development  of  an 
aceurate  appreeiation  of  user  behavior,  traffie  volume  statisties  and  a  ROIMI.  Many 
high-end  Web  analytie  features  (e.g.,  session  analysis,  multiple-site  aggregation)  offer 
online  analytieal  proeessing  (OLAP)  and  data  mining  functionality  via  Java-scripted  Web 
pages  to  eollect  data.  This  relatively  new  teehnology  provides  signifieantly  more 
information  and  seales  better  than  proeessing  of  log  files.  One  obstacle  to  seeing  this 
through  is  that  Java  Server  page  teehnology  doesn’t  work  (yet)  with  Active  Server  Page 
portlets  -  that  means  portal  providers  or  integrators  must  adapt  eontent  to  a  format  the 
portal  understands  sueh  HTML,  XML  or  Wireless  markup  language.  Nevertheless,  work 
to  see  how  OLAP  related  teehnology  ean  be  implemented  into  the  lEEM  metries  analysis 
is  highly  desirable  as  it  would  serve  as  an  enabler  to  deepen  the  analysis  and  shorten 
deeision  and  eorreetive  aetion  proeesses. 

As  the  find  paradigm  of  the  Web  shifts  from  search  to  match  (intelligent  queries 
on  a  query  so  not  to  be  under  or  over  speeified),  new  teehniques  will  be  required  to 
ensure  the  proper  metries  are  used  to  monitor  which  content  nuggets  are  sought  from 
where  and  to  gauge  the  impaets  on  the  user  experienee.  A  promising  teehnique  that 
should  be  explored  to  assist  in  this  endeavor  is  multivariate  clustering-,  a  statistieal 
teehnique  for  dimensionality  reduetion  and  eluster  analysis  applied  to  develop  groups  of 
similar  online  users  based  on  eommonly  held  value  eharaeteristies  from  among  a  baseline 
of  value-driven  variables/metrics.  This  technique  explores  different  solutions  eluster 
(and  sub-cluster)  baseline  solutions.  It  should  be  further  researehed  to  see  if  it  eould 
serve  as  a  viable  trial  for  subsequent  ehange  reeommendations  based  on  the  previous 
metrics  results.  This  type  of  researeh  and  analysis  ean  present  more  insightful  and 
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diverse  data  in  a  shorter  period  of  time  with  less  disruption  to  the  organization.  As  a 
result,  the  information  gathered  from  this  teehnique  may  lead  to  quieker,  more  aceurate 
adjustments  to  the  value  azimuth  indieator  of  lEEM  and  eventually  to  eomplex 
algorithmie  equations  needed  in  the  software  engineer  eommunity  which  reflect  the 
dynamics  of  the  quality  paradigm  taking  place  in  all  segments  and  constituents  that 
underscore  value  in  corporate  portal  processes. 

Domain  analysis  for  the  Intranet  is  never  completely  finished.  Product  definitions 
evolve  continuously.  The  development  of  a  particular  system  that  exploits  previously 
accumulated  domain  knowledge  can  be  the  source  for  new  insights  about  the  domain  that 
adds  to  or  refines  codified  domain  knowledge.  As  the  multiple  uses  of  Web  analytics 
become  apparent,  new  correlations  will  be  blended  to  create  unique  value  for  each 
consumer  of  the  information.  There's  no  one  metric  that  is  right  for  all  companies  in  all 
circumstances  all  the  time.  Inevitably,  however,  all  measurements  will  continue  to  strive 
toward  the  quantification  of  benefits  wherever  possible.  The  role  of  software  engineers 
in  this  endeavor  will  be  to  determine  how  best  to  ascertain  value  where  it  is  not  visible 
through  domain  analysis  identification  and  implementation.  lEEM  provides  a  step  in  this 
direction  as  it  aligns  metrics  to  business  priorities  and  intranet  IT  initiatives  as  well  as  the 
most  relevant  factors  within  all  six  underlying  segments  which  have  the  most  impact  on 
deriving,  measuring  and  increasing  value.  As  portals  become  as  ubiquitous  in 
corporations  as  email  is  today,  more  inter-disciplinary  research  needs  to  be  devoted  in 
this  area  to  help  organizations  of  all  types  and  sizes  recognize,  measure  and  capitalize  the 
full  potential  of  their  greatest  assets:  customer  data  and  intellectual  knowledge. 
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APPENDIX  A:  CONSTITUENTS  TO  FINDING  INTRANET  INFORMATION 


PROBLEM: 

Knowledge  workers  eannot  diseover  all  of  the  information  they  need  to  do  their  jobs 
better. 

QUESTION: 

What  makes  information  more  diseoverable? 

REQUISITES: 

•  Amount  of  information  and  meta-information  (aeeessibility  &  manageability) 

•  Understanding  information  seeking  and  use  behavior 

•  Navigation  system  design  used  to  expose  information  (logieally  grouped) 

•  Confidenee  in  quality  of  information  and  meta-information 

•  Relevaney  to  knowledge  worker 

CONCEPT: 

“Diseoverabibty”:  -  Eaeilitate  information  diseovery  through  presentation  of 
relative  assoeiations  whieh  lead  to  better  deeisions,  inereased  produetivity  and 
effeetiveness  that  would  not  have  been  likely  otherwise.  Diseoverabibty  requires  three 
events  to  happen:  1)  assoeiative  eonneetions  between  eontent  items  are  ereated;  2)  the 
information/  knowledge  is  presented  to  illustrate  relative  assoeiations  that  ean  be  aeted 
upon;  and  3)  the  knowledge  worker  has  the  eognitive  ability/reeognizes  there  is  an  aetion 
to  be  taken. 

CONSTITUENTS: 

Diseoverabibty  in  an  information  system  sueh  as  a  eorporate  intranet  is  predieated 
and  realizable  if  usable  relative  associations  semantieally  exist.  The  term  relative 
associations  eneompasses  all  types  of  relationships  between  items  in  an  information 
system.  These  relationships  are  based  on  sueh  things  as  oharaeteristies  of  the  items 
(properties),  knowledge  worker  tasks  and  interaetions  between  the  items.  Onee 
established,  these  assoeiations  ean  be  exposed  and  leveraged  in  an  information  system  to 
enable  a  knowledge  worker  to  move  from  eoneept  to  eoneept. 

Thus  any  portal  desiring  to  be  part  of  the  information  system  should  eontain  the 
following  13  eonstituents  of  diseovering  information  that  eolleetively  sustain  the 
requisites  above  and  render  relative  assoeiations  useful.  These  eonstituents  are  grouped 
below  into  two  eategories  or  domains:  “Baek-end”  -  the  proeesses  that  take  plaee  that  a 
knowledge  worker  does  not  see;  and,  “Eront-End”  -  the  proeesses  in  whieh  a  knowledge 
worker  has  a  more  direet  interfaee. 
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CONSTITUENTS 

BACK-END 


INSTANCES  OF 


EXPLANATION 


Content  and  Content 
Properties 

People,  Organization, 
Tools,  Application 
(scope  is  beyond 

documents) 

Covers:  Search,  Browse 
&  Data  Mgmt 

Content  -  All  documents  —  help  text,  search  results, 
forms,  or  application  information  —  that  is  delivered  to  a 
knowledge  worker  via  a  website,  or  application.  It  is 
both  subject  and  specific  information.  For  example, 
“People”  is  a  subject  and  a  person  by  name  is  specific 
information.  Content  is  organized  and  found  depending 
on  its  properties  and  associations 

Content  Properties  -  How  content  is  described  in  order 
to  affect  the  associations  made  in  information  retrieval 
and  presentation.  The  characteristics  of  a  content  item 
make  up  its  properties,  such  properties  (a.k.a.  attributes) 
include  author,  length,  name,  etc.  If  content  is  to  be 
discoverable,  it  requires  management  of  its  properties 
and  associations,  as  well  as  the  tools  and  applications 
used  to  retrieve  and  present  them.  How  well  this  is 
managed  affects  the  overall  effectiveness  of  any  method 
of  discovery,  such  as  search  and  browse. 

Domain  Integration 
Framework 

Domain  Ownerships, 

Relationships,  “Key 

Sites” 

Representation  of  the  relationships/  structure  between 
the  key  elements  within  a  design.  It  shows  how  these 
elements  interact  and  must  “transfer”  information,  so 
that  the  architect  can  build  the  environment.  Examples 
are  information  maps  (document  elements),  server 
topography,  scope  of  services,  permissions  (ownership). 

Information  Life 

Cycle 

Expiring,  Adding, 

Versioning,  Retention 

The  events  often  repeated  again  and  again  in 
maintaining  the  relevance  and  accessibility  of  content  in 
an  information  system.  These  events  include  updating, 
versioning,  archiving  and,  when  necessary,  deleting. 
(See:  Retention  in  glossary). 

Search 

UI  (Process  and 

Presentation),  Crawl 

An  application  employed  by  knowledge  workers  as  a 
tool  to  find  through  direct  surfacing  or  through  surfacing 
an  obvious  navigational  path. 

Taxonomy 

Vocabulary,  Hierarchy, 
Meta-Data,  Schemas, 
Heuristics  from 

Longhorn? 

The  result  of  identifying,  creating  and  naming  relative 
associations  between  items  in  an  information  system.  In 
turn,  the  relative  associations  are  grouped  or  classified 
into  taxonomic  structures  based  on  three  relationship 
types:  hierarchical,  equivalence  and  associative. 
Examples  are  controlled  vocabularies;  metadata  scheme; 
category  labels.  (See:  Relative  Associations  in 

glossary). 

User  Data 

“My  Data”, 

“My  Profile”, 

Personalization, 
Segmentation,  Metrics, 
Query  Logs 

Facts  and  figures  a  knowledge  worker  maintains  private 
access  to  for  knowledge  retention  and  expansion.  This 
is  also  referred  to  as  “personalization”  (see  below). 

Table  9,  Back-End  Constituents 
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CONSTITUENT  INSTANCES  OF  EXPLANATION 


FRONT-END 

Accessibility 

Point  of  Access, 

Packaging,  Formatting, 
Presentation 

Information  is  considered  accessible  when  it  is 
available,  reachable,  and  understandable.  (Assuming  the 
information  exists,  ideally  availability  could  occur  from 
any  point  of  access  of  and  point  of  entry  to  the 
information  system.  Although  how  reachable 

information  is  depends  on  a  number  of  factors,  in  this 
context  the  more  information  is  compatible  with  a 
shared  Information  Architecture,  the  easier  it  is  to 
retrieve.  How  this  information  is  then  packaged  and 
presented  in  a  fashion  that  can  be  understood  by  the 
knowledge  worker  is  the  last  stage  of  accessibility.) 

Communication  of 

Authoritativeness  and 
Importance 

Site  Maps,  Credibility 
(Best  Bet),  Authoritative 
Flag,  Relevancy 

Communicating  to  the  knowledge  worker  the  credibility 
of  an  information  system  within  that  system  to  ensure 
confidence  and  trust.  See:  Authoritative  Flags. 

Communication  of 

Understanding 

Links,  Definitions, 

Mouse-over,  Tool  Tips, 
Contacts,  Smart  Tags, 
Context 

Communicating  to  the  knowledge  worker  meaning  and 
significance  to  ensure  acceptance  and  engagement, 
which  further  increases  the  audience’s  propensity  to  act 
on  information  in  a  common  way,  i.e.,  the  Glossary  of 
IP  Terms  and  Definitions,  Mouse-over,  and  best 
practices. 

Information  Grouping 
&  Segmenting 

Content  Nuggets, 

Categories,  Applications 

Information  grouping  is  the  logical  collection  of  similar 
and  relevant  information,  such  as  Content  Nuggets  and 
Categories  respectively.  Another  aspect  of  this  is 
information  segmenting  in  which  only  the  relevant  parts 
of  a  document,  i.e.  a  paragraph  or  two  are  extracted  and 
placed  into  fragmented  groupings  instead  of  the  entire 
document.  For  example,  first  200  words  of  a  document 
returned  as  part  of  a  search  result. 

Navigation 

-  Local 

-  Global 

Presentation 

Consistent  Labeling 
Sources  of  information, 
applications 

Method  of  moving  through  the  domain  framework  by 
way  visual  presentation  and  consistent  choices. 
Navigation  has  2  basic  types:  local  (also  known  as 
vertical)  and  global  (also  known  as  horizontal). 

Local  navigation  presents  choices  leading  to  subtopics 
or  sub-areas  of  a  site,  usually  defined  by  one  of  its  main 
menu  subjects  (often  referred  to  as  a  “drill  down”). 

Global  navigation  presents  choices  leading  to  other  main 
areas  of  a  site,  i.e.  Home;  Search;  About.  Global 
navigation  is  consistent  throughout  an  information 
system  which  allows  knowledge  workers  to  go  across 
portals  with  a  sense  of  familiarity. 

Personalization  and 
Customization 

Security,  Content 

Filtering, 

Authentication, 

Personalization  -  A  method  of  contextualizing 
information  for  a  knowledge  worker  based  on  what  is 
known  about  the  knowledge  worker.  A  site  can  use 
personalization  to  alter  navigation  and  content  presented 
according  to  the  perceived  needs  of  the  knowledge 
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worker.  Examples  include  content  filtering  based  on 
role;  authentication  based  on  name  or  status  as  a 
manager. 

Customization  -  Ability  for  knowledge  workers  to  self- 
configure  the  contents/  constituents  of  their  domain 
site.  In  essence,  individuals  are  allowed  to  subscribe  to 
constituents,  or  sub-eonstituents  of  a  portal,  including 
navigation,  content,  search,  etc.,  in  order  to  provide  and 
store  information  (i.e.,  like  the  notion  of  “my  saved 
searches”)  most  relevant  to  their  interests,  position  or 
role. 

User  Assistance 

Help  in  context, 

Feedback,  Dialog  with 
the  knowledge  worker. 
Training 

Help  made  available  to  the  knowledge  worker  while 
using  an  information  system.  User  assistance  provides 
guidance  on  how  to  use  the  system  or  additional  help  in 
finding  information  sought.  Examples  are  help  in 
context,  feedback.  Dialog  boxes,  and  training,  real  time 
assistance  from  a  human  intermediary  via  a  live  chat. 

Table  10.  Front-End  Constituents 
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APPENDIX  B:  ONLINE  E-SURVEY  EXAMPLE 


Useful  general  e-survey  questions  about  intranets  and  employee  portals  inelude: 

One  minute  of  your  valuable  input  will  help  the  company  provide  you  with  better  intranet 
service.  Please  take  a  moment  to  complete  this  automatic  survey  as  part  of  our  on-going 
effort  to  optimize  out  intranet  according  to  your  needs: 

1-  means  seldom,  very  little,  poor,  no 
5-  means  very  often,  very  much,  exception,  yes 


What  is  the  most  useful  feature  of  the  portal  for  you?  (To  be  chosen  from 
provided  list  of  portal  features.) 

A. ) 

B. ) 

C. ) 

Which  of  the  following  processes  do  you  use  the  portal  to  accomplish? 
(To  be  chosen  from  provided  list  of  portal  processes.) 

A. ) 

B. ) 

C. ) 


1-2-3 -4-5  Which  new  feature  would  increase  your  portal  usage  the  most?  (To  be 
chosen  from  menu  of  potential  new  portlets.) 

A. ) 

B. ) 

C. ) 


How  can  the  portal  be  improved  to  enable  employees  to  be  more 

productive? 


1-2-3-4-5 


1-2-3-4-5 


1-2-3-4-5 


1-2-3-4-5 


How  critical  is  the  portal  to  performing  your  job  function? 
How  up-to-date  is  the  information  in  the  portal? 

How  would  you  rate  the  performance  of  the  portal? 

Has  the  portal  met  or  exceeded  your  expectations? 


1-2-3-4-5  How  relevant  is  the  information  in  the  portal  to  your  job? 


1-2-3-4-5 


1-2-3-4-5 


1-2-3-4-5 


Is  the  portal  part  of  your  daily  routine? 

How  would  you  rate  ease  of  use  of  the  portal? 

How  many  times  per  day  (or  what  percentage)  do  you  use  the  portal? 


SUBMIT 

THANK  YOU 

(Note:  To  be  useful,  the  survey  must  capture  timing  and  demographic  information 
about  the  respondent  (see  Figure  2,  Intranet  Domains  and  Segments).  When  this 
information  cannot  be  determined  automatically  from  a  respondent’s  profile,  the 
respondent  should  be  queried  directly.  ) 
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APPENDIX  C:  METRICS  TO  MEASURE  INTRANET  PERFORMANCE 


The  following  table  lists  numerous  specific 

Revenue  Eorecast 

metrics  that  can  be  applied  to  measure  a 

Qualified  Revenue 

portal  site’s  activity  -  not  all  of  them  are 

non-Qualified  Revenue 

needed  for  an  intranet.  When  a  subset  of 

Visits 

these  metrics  are  further  logically 

Qualified  Visits 

represented  and  grouped  in  agreement  with 

non-Qualified  Visits 

the  segments  and  constituents  that  the 

Product  Qualifications 

lEEM  outlines  in  support  of  assessing 

Visitor  Conversion  Ratio 

critical  business  requirements,  rational 

Results  by  Visitor  Qualification 

subsequent  and  impactful  actions  can  be 

Results  by  Product  Category 

taken  based  off  these  metric  results  which 

Results  by  Product 

lead  to  coherent  and  sensible  value  gains 

Cost 

(see  Appendix  D  for  the  baseline  approach 

Summary 

to  do  this). 

Per  Visit 

-represents  a  metric  category 

Per  Qualified  Visit 

Grey  -represents  a  useful  lEEM  metric 

Per  non-Qualified  Visit 

Per  Product  Qualification 

ROI 

General  Statistics 

Summary 

Page  Views  Over  Time 

Per  Day 

Top  Pages  by  Visits 

Per  Visit 

Advertising  Click  Through  Rate 

Per  Qualified  Visit 

Hits  Over  Time 

Per  non-Qualified  Visit 

Visitors  Over  Time 

Per  Product  Qualification 

Top  Visitors 

Projections 

Top  Referring  Sites  by  Visits 

Revenue  Eorecast 

Top  Browsers  by  Visits 

ROI 

Commerce  Analysis 

Visit 

Summary 

Qualified  Visit 

Executive  Summary 

non-Qualified  Visits 

Results  Summary 

Product  Qualifications 

Abandonment  Summary 

Referrers 

Results  by  Marketing  Campaign 

Revenue  Eorecast 

Shoppers 

Qualified  Revenue  Eorecast 

Buyers 

non-Qualified  Revenue  Eorecast 

Attempted  Buyers 

Visits 

Engagement  Rate 

Qualified  Visits 

Conversion  Rate 

non-Qualified  Visits 

Abandonment  Rate 

Product  Qualifications 

Summary  by  Marketing  Campaigns 

Results  by  Visitor  Qualification 

Marketing  Campaigns 

Results  by  Product  Category 

Results 

Results  by  Product 

Summary 

Product  Categories 
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Table  9  -  Specific  Metrics 


Revenue  Forecast 

Product  Qualifications 

Results  by  Product 

Results  by  Visitor  Qualification 

Results  by  Marketing  Campaign 

Results  by  Referrer 

Products 

Revenue  Forecast 
Product  Qualifications 
Results  by  Visitor  Qualification 
Results  by  Marketing  Campaign 
Results  by  Referrer 
Products  Summary 


Top  Pages 

Top  Pages  by  Visits 

Top  Pages  by  Visits  Over  Time 

Top  Pages  by  Views 

Top  Pages  by  Views  Over  Time 

Top  Documents 

Top  Documents  By  Visits 

Top  Documents  By  Visits  Over  Time 

Top  Documents  By  Views 

Top  Documents  By  Views  Over  Time 

Dynamic  Pages  &  Forms 

Dynamic  Pages  &  Forms  by  Visits 

Dynamic  Pages  &  Forms  by  Visits  Over 

Time 

Dynamic  Pages  &  Forms  by  Hits 
Dynamic  Pages  &  Forms  by  Hits  Over 
Time 

Page  Views  Over  Time 

Top  Content  Groups 

Top  Content  Groups  by  Visits 

Top  Content  Groups  by  Visits  Over  Time 

Top  Content  Groups  by  Hits 

Top  Content  Groups  by  Hits  Over  Time 

Top  Entry  Pages 

Top  Entry  Pages  by  Visits 

Top  Entry  Pages  by  Visits  Over  Time 

Eeast  Requested  Entry  Pages 

Eeast  Requested  Entry  Pages  by  Visits 

Eeast  Requested  Entry  Pages  by  Visits 

Over  Time 

Top  Exit  Pages 

Top  Exit  Pages  by  Visits 


Top  Exit  Pages  by  Visits  Over  Time 

Single  Access  Pages 

Single  Access  Pages  by  Visits 

Single  Access  Pages  by  Visits  Over  Time 


Top  Paths  Through  Site  by  Visits 
Top  Destination  Paths  Through  Site 


Hits  Over  Time 

Top  Directories 

Top  Directories  by  Visits 

Top  Directories  by  Visits  Over  Time 

Top  Directories  by  Hits 

Top  Directories  by  Hits  Over  Time 

Top  Directories  by  Kbytes  Transferred 

Most  Downloaded  Piles 

Most  Downloaded  Piles 

Most  Downloaded  Piles  Over  Time 

Most  Downloaded  Piles  by  Visits 

Most  Downloaded  Piles  by  Visits  Over 

Time 

Most  Accessed  Pile  Types 

Most  Accessed  Pile  Types 

Most  Accessed  Pile  Types  by  Kbytes 

Transferred 

Most  Uploaded  Piles 

Most  Uploaded  Piles 

Most  Uploaded  Piles  by  Visits 

Top  Entry  Piles 

Top  Entry  Piles  by  Visits 

Top  Entry  Files  by  Visits  Over  Time 

Eeast  Requested  Entry  Piles 

Least  Requested  Entry  Piles  by  Visits 

Least  Requested  Entry  Piles  by  Visits 

Over  Time 


URL  ID  Parameter  Analysis  by  Visits 
URL  ID  Parameter  Analysis  by  Hits 
URL  2D  Parameter  Analysis 


Advertising  Click  Through  Rate 
Ad  Visits 
Ad  Visits 

Ad  Visits  Over  Time 
Ad  Views 
Ad  Views 
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Table  9  -  Specific  Metrics 


Ad  Views  Over  Time 
Ad  Visits  With  Clicks 
Ad  Visits  With  Clicks 
Ad  Visits  With  Clicks  Over  Time 
Ad  Clicks 
Ad  Clicks 

Ad  Clicks  Over  Time 


Top  Visitors 

Top  Visitors 

Top  Visitors  Over  Time 

Top  Visitors  by  Hits 

Top  Visitors  by  Hits  Over  Time 

New  vs.  Returning  Visitors 

New  vs.  Returning  Visitors 

New  vs.  Returning  Visitors  Over  Time 

Top  Authenticated  Visitors 

Top  Authenticated  Visitors 

Top  Authenticated  Visitors  Over  Time 

Top  Authenticated  Visitors  by  Hits 

Top  Authenticated  Visitors  by  Hits  Over 

Time 

Visitors  Over  Time 
Visitors  Over  Time 
Visits  Over  Time 
Visitors  by  Number  of  Visits 


Top  Geographic  Regions 

Most  Active  Countries 

North  American  States  and  Provinces 

Most  Active  Cities 

Most  Active  Organizations 

Most  Active  Organizations  by  Visits 

Most  Active  Organizations  by  Hits 

Top-Level  Domains  Types 

Top-Level  Domains  Types  by  Visits 


Summary  of  Activity  for  Report  Period 
Visits  by  Number  of  Pages  Viewed 
Bandwitdh:Kbytes  Transferred  Over  Time 
Average  Time  to  Serve  Documents 
Average  Time  to  Serve  Dynamic  Pages 
Day  of  Week 
Visits  by  Day  of  the  Week 
Hits  by  Day  of  the  Week 


Hour  of  Day 

Visits  by  Hour  of  the  Day 
Hits  by  Hour  of  the  Day 
Length  of  Visit 
Visits  by  Length  of  Visit 
Page  Views  by  Length  of  Visit 


Server  Cluster  Load  Balance  by  Hits 
Server  Cluster  Load  Balance  by  Kbytes 
Transferred 


Technical  Statistics  and  Analysis 
Dynamic  Pages  &  Forms  Errors 
Client  Errors 
Server  Errors 


Top  Referring  Sites  by  Visits 
Top  Referring  UREs  by  Visits 
Top  Search  Engines 
Top  Search  Phrases 
Top  Search  Keywords _ 


Top  Browsers 

Top  Browsers  by  Visits 

Top  Browsers  by  Visits  Over  Time 

Top  Browsers  by  Hits 

Top  Browsers  by  Hits  Over  Time 

Microsoft  Explorer  Browsers 

Microsoft  Explorer  Browsers  by  Visits 

Microsoft  Explorer  Browsers  by  Visits 

Over  Time 

Microsoft  Explorer  Browsers  by  Hits 

Microsoft  Explorer  Browsers  by  Hits 

Over  Time 

Netscape  Browsers 

Netscape  Browsers  by  Visits 

Netscape  Browsers  by  Visits  Over  Time 

Netscape  Browsers  by  Hits 

Netscape  Browsers  by  Hits  Over  Time 

Top  Platforms 

Top  Platforms  by  Visits 

Top  Platforms  by  Visits  Over  Time 

Top  Platforms  by  Hits 

Top  Platforms  by  Hits  Over  Time _ 


Table  11,  Specific  Metrics 
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APPENDIX  D:  INTRANET  PORTAL  METRIC  BREAKDOWN 


The  ensuing  tables  of  baseline  metrics  and  conversion  ratios  is  color  coded  in  line  with 
the  illustration  of  the  model  below  to  highlight  from  which  domain  and  metric  type  metrics 
originate.  The  model  is  an  approach  to  determining  effectiveness  taking  into  account  the 
pertinent  characteristics  of  intranet  domains,  segments  and  constituents  with  the  resulting 
supposition  delivered  by  a  variety  of  metrics  specified  in  priority  and  divided  into  audiences  and 
responsibilities  according  to  business  requirements. 


Front-End 


Easier  to  Measure 


Back-End 


Sat.  Surveys 

-UE 

No.  Clicks  to  Find 
Tie  to  Locate 


Activity  by 

llcanA  Frequent 
Time,  &  usage 

Duration  Traffic  Volume 


Unique  Users 


People,  Process,  Technology 


Corresponding 
Color  Codes  on 
Metric  Baseline 
Tables 


Business  Req.  or 
Derived  Metric 

Back  End  or 
Hard  Metric 

Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


Figure  18,  lEEM  Color  Code  Supporting  Metrics  Baseline  Tables 
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Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 

Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


WHY: 

HOW: 

< - 

— business  process - 

- X - best  practices  — 

< - 

—  lEEM  Related 

Business  Issue: 

-  Metric  Area 

Business  Question  of 
Web  Site  Activity 

Business  Significance  of 
Web  Site  Report  Solution 

Soecific  Sub-Metric  Area 

-  Specific  Metric 

Who  Benefits  and  Why 
(User,  Portal,  Enterprise) 

lEEM  Seementts): 

lEEM  Constituent(s) 

Efficiency  (E) 
or 

EfTectiveness(F) 

Suceess: 

-  Visitors  (by 
Search  Engine) 

-  Downloads 
(by  Search 
Engine) 

Which  configuration 
for  our  search  engines 
should  I  pay  attention 
to? 

How  a  chosen  search 
engine  is  configured  takes 
time  and  money  to 
develop.  Configurations 
must  be  justifiable  in  terms 
of  relevant,  helpful  finds 
and  downloads  or 
takeaways,  which  lead  to 
better  decisions  and 
opportunity,  leading 
ultimately  to  increased 
revenue  and  understanding 
the  ROI.  Thus 
determining  which 
configuration  (including 

Best  Bets  and  Best 

Practices)  provides  the 
best  relevance  is  critical  to 
enabling  users  to  find  the 
information  they  need  to 
do  their  jobs. 

Visitor  bv  Search  En2.  Confi2uration 

-  Top  Referring  Sites  by  Visits 

-  Top  Documents  By  Visits 

-  Top  Content  Groups  by  Visits 

-  Top  Entry  Pages  by  Visits 

-  Top  Directories  by  Visits 

Downloads  bv  Search  Engine 

-  Top  Documents  By  Views 

-  Top  Content  Groups  by  Hits 

-  Top  Directories  by  Kbytes  Transfer 

-  Top  Search  Engines 

ROI  bv  Search  Engine  Configuration 

-  Best  Bet:  Most  relevant  hits 

-  Successful  Approach 

-  User  Survey:  User  Feedback 

User  -  Find  more  frequently  infonnation  they  need 

Portal  -  Product  Manager  provides  suggestions  (in  the 
form  of  collective  Best  Bets  and  Best  Practices)  to 
search  configuration  algorithms 

Enterprise  -  When  users  and  portals  benefit  from  more 
relevant  search  hits,  so  does  the  enterprise  in  aggregate 

Info.  Architecture 

-  Search 

F 

WHAT:  RELEVANCE  (Priority  No.  1) 

WHERE:  Enterprise 

WHO:  Portal  Owner  Related 


Table  12,  Relevance  Metrics  to  Business  Requirements 
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WHAT:  TRAFFIC  VOLUME  (Priority  No.  2) 

WHERE:  Enterprise 

WHO:  Portal  Owner  Related 

WHY:  HOW: 


Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 

Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


< - BUSINESS  PROCESS 


>< - best  practices - > 


< - lEEM  Related - > 


Business  Issue: 

-  Metric  Area 


Business  Question  of  Business  Significance  of 
Web  Site  Activity  Web  Site  Report  Solution 


Specific  Snb-Metric  Area 

-  Specific  Metric 


Wbo  Benefits  and  Wby 

(User,  Portal,  Enterprise) 


lEEM  Segmentts): 

lEEM 

Constituent(s) 


Efflciency  (E) 
or 

Effectiveness(F) 


Usefulness 
(Leads  Commit. 

and  Completed): 

-  VaM 

- 

-  Loycilt^i 


How  many  visitors  is  my 
web  site  producing?  Are 
we  converting  visitors  to 
committed  leads,  i.e.,  an 
exchange  of  information, 
and  completed  processes, 
i.e.,  downloads? 


Identify  and  track  leads  by 
web  browsing  behavior  to 
understand  total  quantity 
and  quality  of  leads. 
Quantify  lead  to  download 
conversion  rates  (or  length 
of  site  visit  duration)  to 
understand  lead  generation 
effectiveness  in  order  to 
better  understand  what 
adjustments  need  to  be 
made  (i.e.,  site  re-design, 
elevation  of  highly  sought 
after  pages  or  click  to 
success  and  pages  to  drop). 


Visitors  to  Leads 

-  Top  Pages  by  Views  Over  Time 

-  Top  Documents  by  Views  Over  Time 

-  Page  Views  by  Length  of  Visit 

-  Visits  by  Length  of  Visit 

-  Top  Exit  Pages  by  Visits 

Total  Leads 

-  Most  Accessed  File  Types 

-  Most  Uploaded  Files 

-  Top  Directories  by  Hits  Over  Time 

-  Top  Content  Groups  by  Hits  Over 
Time 

Lead  to  Download  Conversion 

-  Most  Downloaded  Files  by  Visits 

-  Most  Downloaded  Files  by  Visits  Over 
Time 


User  -  Exchange  information  only  when  necessary 
and  download  or  view  (time  duration)  what  is 
needed. 


Info  Architecture 
-  Domain  Integrated 
Framework  (DIF) 


Portal  -  Managers  learn  where  visitors  fail  to  make 
leads  or  where  leads  fail  to  complete  the  process  and 
can  make  changes  accordingly  (analysis  can  be 
provided  by  Product  Manager) 


Design 

-  Usability  Study 


Enterprise  -  When  users  and  portals  benefit  from 
visits  becoming  leads  and  leads  being  completed  in 
the  form  of  downloads  or  views,  the  enterprise 
benefits  as  well  by  virtue  of  users  making  better 
decisions  from  finding  what  they  need. 


F 


Usability  Study 

-  Usability 


Table  13.  Traffic  Volume  Metrics  to  Business  Requirements 
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WHAT: 

UNIQUE  USERS  (Prioritv  No.  3) 

WHERE: 

Enterprise 

WHO: 

Portal  Owner  and  Portal  Manager  Related 

WHY: 

HOW: 

Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 

Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


< - BUSINESS  PROCESS - >< - BEST  PRACTICES - >  < - lEEM  Related - > 


Business  Issue: 

-  Metric  Area 

Business  Question  of 
Web  Site  Activity 

Business  Significance  of 
Web  Site  Report  Solution 

Snecific  Metric  Area 

-  Specific  Metric 

Who  Beneflts  and  Why 
(User,  Portal,  Enterprise) 

lEEM  ScemcntlsI: 

lEEM  Constituent(s) 

Efficiency  (E) 
or 

ElTectivenessfF) 

Retention: 

How  effectively  am  I 

Determine  how  quickly 

Return  Visitor  Rate 

User  -  Gains  familiarity  to  a  site  which  reduces  overall 

Info  Architecture.: 

- 

Eov«/oj 

VahiJi 

building  loyalty  with 
my  visitors? 

you  are  building  your  user 
base  to  gage  site  audience 
enlargement  and  shrinkage 
over  time  and  vis  a  vis 
other  sites. 

-  Top  Visitors  (authenticated) 

-  Visitors  by  Number  of  Visits 

-  Visitors  Over  Time 

-  Top  Visitors  by  Hits  (Leads) 

-  Top  Visitors  by  Hits  Over  Time 

-  Top  Visitors  by  Leads  Completed 

-  Visits  by  Length  of  Visits 

*  All  of  the  above  for  Returning  Visitors 

-  Hew  ys.  Returning  Visitor 

frustration. 

Portal  -  Managers  learn  where  return  visitors  come 
from  and  how  many  there  are,  which  a  partial  indication 
that  users  find  the  site  helpful. 

Enterprise  -  When  users  continue  to  return  to  the  same 
site  for  information,  it  indicates  that  they  find  it  helpful 
and  s  well  as  where  the  sources  of  richness  lie. 

-User  Data 

F 

Ontimization: 

What  do  my  visitors 

Analyze  the  most  popular 

Return  Visitor  Tarset  Paecs 

User  -  Popular  downloaded  information  remains  as  well 

Info.  Knowledge 

LoyaJM 

Valitr 

tgilm 

^timizatio4 

come  back  for? 

content  for  my  return 
visitors  in  order  to  load 
and  associate  related 
information  to  meet 
demand. 

(correlate  return  visits  with  content): 

-  Top  Returning  Visitors  by  Hits 

-  Top  Returning  Visitors  by  Hits  Over 
Time 

-  Top  Returning  Visitors  by  Leads 
Completed 

-  Returning  Visitor  Visits  by  Length  of 
Visits 

-  Returning  Visitor  Page  Views  Over 

Time 

-  Top  Document  and  Content  Group  for 
Returning  Visitors  Over  Time 

Survey 

-  Internal  Returning  Visitor  Session 
Activity 

as  other  information  like  it  eventually  being  posted  or 
better  associated. 

Portal  -  Managers  learn  which  pages  are  desired  and 
can  load  related  information  to  meet  demand. 

Enterprise  -  As  portals  more  accurately  monitor  what 
information  is  sought  and  add  additional  information 
that  is  related,  the  enteiprise  is  doing  a  better  job 
meeting  the  expectations  and  business  needs  of  its  users. 

-  Survey 

Info  Architecture 

-  Info  Grouping 

-  User  Data 

F 

Table  14,  Unique  User  Metrics  to  Business  Requirements 
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WHAT: 

UNIQUE  USERS  tPriorltv  No.  3) 

WHERE: 

Enterprise 

WHO: 

Portal  Owner  and  Portal  Manager  Related 

WHY: 

HOW: 

Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 


Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


< - BUSINESS  PROCESS 


>< — best  practices - > 


< - lEEM  Related - > 


Business  Issue: 

-  Metric  Area 


Business  Question  of  Business  Significance  of 
Web  Site  Activity  Web  Site  Report  Solution 


Specific  Snb-Metric  Area 

-  Specific  Metric 


Who  Benefits  and  Why 

(User,  Portal,  Enterprise) 


lEEM  Segmentts):  Efflciency  (E) 

lEEM  Constituent(s)  ElTectiv°ene..(F) 


Targeted; 

-  Itc^cA 

-  Loyalj^ 

-  ValiiJi 

-  Disintermedi¬ 
ation 

-  Access 


Where  do  my  visitors 
come  from 
(specifically,  which 
regions,  organizations, 
groups,  roles)? 


Identify  where  users 
originate  and  how  your 
web  site  is  searched  and 
browsed  to  understand 
what  content  areas  are 
most  effective  to  improve 
overall  productivity  for  a 
particular  region 
organization,  group  and/or 
role). 


Users  by  Region 

-  Most  Active  by  Visits 

-  Most  Active  by  Hits 

-  Most  Returning  Visitors 

Users  by  Group 

-  Most  Active  by  Visits 

-  Most  Active  by  Hits 

-  Most  Returning  Visitors 

Users  by  Role 

-  Most  Active  by  Visits 

-  Most  Active  by  Hits 

-  Most  Returning  Visitors 


User  -  Potentially  better  personalized  service 

Portal  -  Managers  learn  which  pages  are  desired  and 
most  effective  by  region,  organization,  group  and  role 
and  can  make  enhancements  accordingly. 

Enterprise  -  As  portals  more  accurately  monitor  what 
information  is  sought  and  add  additional  information 
that  is  related,  the  enterprise  is  doing  a  better  job 
meeting  the  expectations  and  business  needs  of  its  users, 
both  enterprise  and  world  wide. 


Info.  Knowledge 

Info  Architecture: 

-DIF 

-Taxonomy 
-Content  Properties 
-Accessibility 

Ed.  Programming 

-  Usability 

-  Ease  of  Navigation 


F 


Usability  Study 

-  Internal  Visitor  Session  Activity 

-  Usability 


Table  14,  Unique  User  Metrics  to  Business  Requirements 
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WHAT:  CROSS  PORTAT.  REFERRAT.  (Priority  No.  4) 

WHERE:  Enterprise 

WHO:  Portal  Owner  Related 

WHY:  HOW: 


Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 

Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


< - BUSINESS  PROCESS - >< - BEST  PRACTICES - >  < - lEEM  Related - > 


Business  Issue: 

-  Metric  Area 

Business  Question  of 
Web  Site  Activity 

Business  Significance  of 
Web  Site  Report  Solution 

Snccific  Snb-Metric  Area 

-  Specific  Metric 

Who  BeneHts  and  Why 
(User,  Portal,  Enterprise) 

lEEM  Seementls): 

lEEM  Constituent(s) 

Efficiency  (E) 
or 

EfTectiveness(F) 

Aehievement: 

- 

-  Disintermedi¬ 
ation 

Is  my  web  site 
becoming  a  more 
important  channel  for 
information? 

By  measuring  customer 
satisfaction  and  the  visitors 
(as  well  as  subsequent 
leads)  from  others  web 
sites  to  yours,  you  will  be 
able  to  understand  your 
web  site’s  importance  and 
relevance. 

Visitor  bv  Source  (Referral  from  other 

Portal) 

-  Top  Referring  Sites  by  Visit 

-  Top  Referring  URLs  by  Visit 

Leads  bv  Source  (coiTelate  visitor  bv 

source  refen'al  to  lead) 

-  Top  Referring  Sites  by  Visits  to  Leads 

-  Top  Referring  URLs  by  Visits  to  Leads 

Leads  Completed  bv  Source  (correlate 

visitor  bv  source  to  lead  comnleted) 

-  Top  Referring  Sites  by  Visits  to  Leads 
Completed 

-  Top  Referring  URLs  by  Visits  to  Leads 
Completed 

Uniauc  and  New  Users 

-  New  Vi'.  Returning  Visitors 

-  New  Vi.  Returning  Visitors  Over  Time 

Survey 

-  User  Sat. 

User  -  Site  provides  or  directs  you  to  a  site  with  the 
needed  information  and  user  has  opportunity  to  express 
satisfaction. 

Portal  -  Managers  learn  which  pages  are  desired  and 
can  load  related  infonnation  to  meet  demand.  Mangers 
learn  visitor,  lead  and  lead  completed  ratios  by  source  to 
determine  most  valuable  referral  sites. 

Enterprise  -  Ultimately,  the  more  referrals  the  more 
information  is  to  be  discovered.  As  portals  are 
monitored  for  referral  by  source,  the  value  of  a  portal 
takes  on  an  added  dimension  to  overall  significance  or 
enhancer  to  enterprise  information  discovery  (intranet 
portals  will  in  effect  be  both  a  beneficiary  and 
benefactor  of  referrals). 

Info  Architecture: 

-Navigation 

-Search 

Info  Knowledee 

User  Experience 

-  Sat.  Survey 

F 

Table  15,  Cross  Portal  Referral  Metrics  to  Business  Requirements 
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WHAT:  CROSS  PORTAT.  REFERRAL  (Priority  No.  4) 

WHERE:  Portal 

WHO:  Portal  Manager  Related 

WHY:  HOW: 


Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 

Front  End  or 
Soft  Metric 

Best  Bet  or 

Practice 

rracnce 


< - BUSINESS  PROCESS - >< - BEST  PRACTICES - >  < - lEEM  Related - > 


Business  Issue: 

-  Metric  Area 

Business  Question  of 
Web  Site  Activity 

Business  Significance  of 
Web  Site  Report  Solution 

Snccific  Snb-Metric  Area 

-  Specific  Metric 

Who  Benefits  and  Why 
(User,  Portal,  Enterprise) 

lEEM  Sc2mcntfsf: 

lEEM  Constituent(s) 

Efficiency  (E) 
or 

EITectivenesstF) 

Ootimization* 

-  Survey 

How  can  I  maximize 
the  effectiveness  of  my 
online  services 
offerings? 

Measure  the  visitation  to 
online  service  resources 
and  inform  services  people 
of  key  areas  of  concern  or 
need  as  a  mechanism  to 
improve  business 
performance. 

Web  Pa2e  Most  Visited: 

-  Top  Pages  by  Views  Over  Time 

-  Top  Documents  by  Views  Over  Time 

-  Page  Views  by  Length  of  Visit 

-  Visits  by  Length  of  Visit 

-  Top  Exit  Pages  by  Visits 

-  Most  Accessed  File  Types 

-  Most  Uploaded  Files 

-  Top  Directories  by  Hits  Over  Time 

-  Top  Content  Grps  by  Hits  Over  Time 

-  Most  Downloaded  Files  by  Visits 

-  Most  Downloaded  Files  by  Visits 

Over  Time 

Load  Balance  and  Cachin2: 

-  Server  Cluster  Load  Balance  by  Hits 

-  Server  Cluster  Load  Balance  by 

Kbytes  Transferred 

Survey 

-  User  Feedback: 

User  -  Sites  are  proactive  in  monitoring  what  your 
interests  are  in. 

Portal  -  Managers  learn  which  pages  are  desired  and 
can  load  related  information  to  meet  demand. 

Enterprise  -  More  responsive  and  pro-active  in 
providing  overall  Intranet  support,  leading  to  increased 
user  sat. 

Info  Knowlecl2e; 

-  Survey 

Info  Architecture: 

-  User  Data 

F 

Usefulness: 

-  User  Sat. 

^reativit.\i 

Bfc/.v/ort 

Which  user  segments 
utilize  web  service 
pages? 

Anticipate  possible  service 
issues  and  recommend 
solutions  based  on 
segmentation  of  user  type 
and  web  activity. 

User  Se2nient  Usa2e  Rate: 

Users  bv  Grono 

-  Most  Active  by  Visits 

-  Most  Active  by  Hits 

-  Most  Active  by  Duration  of  Visit 

-  Most  Active  by  Download 

-  Most  Returning  Visitors 

User  -  Sites  are  proactive  in  monitoring  what  your 
interests  are  in  (and  provide  some  personalization). 

Portal  -  Managers  learn  user  behavior  and  needs  and 
attempt  to  anticipate  what  information  is  desired  to  the 
lowest  denominator  possible. 

Enterprise  -  More  responsive  and  pro-active  in 
providing  overall  Intranet  support,  leading  to  increased 
user  sat. 

Info  Architecture: 

-  User  Data 

F 

TABLE  15,  Cross  Portal  Referral  Metrics  To  Business  Requirements 
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WHAT:  CROSS  PORTAT.  REFERRAT.  (Priority  No.  4) 

WHERE:  Portal 

WHO:  Portal  Manager  Related 

WHY:  HOW: 


Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 

Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


< - BUSINESS  PROCESS - >< - BEST  PRACTICES - >  < - lEEM  Related - > 


Business  Issue: 

-  Metric  Area 

Business  Question  of 
Web  Site  Activity 

Business  Significance  of 
Web  Site  Report  Solution 

Snecific  Sub-Metric  Area 

-  Specific  Metric 

Who  Benefits  and  Why 
(User,  Portal,  Enterprise) 

lEEM  Scemcntls): 

lEEM  Constituent(s) 

Efficiency  (E) 
or 

Effectiveness(F) 

Preservation: 

-  User  Sat. 

UreatiUm 

-  t^enti^ 

How  do  I  increase  user 
loyalty  by  proactively 
contacting  my  users? 

Deliver  personalized 
services  offerings  (via  UI 
or  email)  directed  toward 
identified  user  needs 

Reneat  Visits 

-  Top  Visitors  (authenticated) 

-  Visitors  by  Number  of  Visits 

-  Visitors  Over  Time 

-  Top  Visitors  by  Hits  (Leads) 

-  Top  Visitors  by  Hits  Over  Time 

-  Top  Visitors  by  Leads  Completed 

-  Visits  by  Length  of  Visits 

*  All  of  the  above  for  Returning  Visitors 

Personalization 

-  Automated  and  User  Determined 
Personalization 

Survey 

-  Emails  to  High  Frequency  Visitors  to 
Solicit  Needs 

User  -  Sites  are  proactive  in  monitoring  what  your 
interests  are  in  (and  provide  some  personalization). 

Portal  -  Managers  learn  user  behavior  and  needs  and 
attempt  to  anticipate  what  information  is  desired  to  the 
lowest  denominator  possible. 

Enterprise  -  More  responsive  and  pro-active  in 
providing  overall  Intranet  support,  leading  to  increased 
user  sat. 

Info  Architecture: 

-  Personalization 

-  User  Data 

User  Exnerience: 

-  Personalization 

F 

-Dintermedia- 

tion 

TABLE  15,  Cross  Portal  Referral  Metrics  To  Business  Requirements 
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WHAT:  CROSS  PORTAL  REFERRAL  (Priority  No.  4) 

WHERE:  Enterprise 

WHO:  Portal  Manager  and  Editor  Related 

WHY:  HOW: 


Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 

Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


< - BUSINESS  PROCESS - >< - BEST  PRACTICES - >  < - lEEM  Related - > 


Business  Issue: 

-  Metric  Area 

Business  Question  of 
Web  Site  Activity 

Business  Significance  of 
Web  Site  Report  Solution 

Snecific  Sub-Metric  Area 

-  Specific  Metric 

Who  Benefits  and  Why 
(User,  Portal,  Enterprise) 

lEEM  Scemcntls): 

lEEM  Constituent(s) 

Efficiency  (E) 
or 

Effectiveness(F) 

Retention: 

-  User  Sat. 

-  Usability 

-  Ease  of 
Navigation 

Why  are  visitors 
leaving  our  site,  even 
though  they  have 
already  selected 
several  items  or  started 
a  lead  session? 

Monitor  dropped  pages 
and  then  study  them  to 
understand  why  they  were 
dropped  and  make 
modifications  (i.e.,  design 
change)  to  improve 
discovery  of  information 

Abandons: 

-  Dropped  page  rate  over  time 

User  Sat.  Survey: 

-  Abandonment  causes 

Usability  Study: 

-  Abandonment  causes 

User  -  Pages  where  users  have  problems  will  be 
identified  and  users  may  have  chance  to  comment  on 
their  modification. 

Portal  -  Managers  learn  which  pages  are  desired  and 
can  load  related  infonnation  to  meet  demand.  Mangers 
learn  visitor,  lead  and  lead  completed  ratios  by  source  to 
determine  most  valuable  referral  sites. 

Enterprise  -  Ultimately,  the  more  referrals  the  more 
information  is  to  be  discovered.  As  portals  are 
monitored  for  referral  by  source,  the  value  of  a  portal 
takes  on  an  added  dimension  to  overall  significance  or 
enhancer  to  enterprise  information  discovery  (intranet 
portals  will  in  effect  be  both  a  beneficiary  and 
benefactor  of  referrals). 

Ed.  Programming. 

User  Exnerience: 

-  Survey/Feedback 

F 

TABLE  15,  Cross  Portal  Referral  Metrics  To  Business  Requirements 
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Intranet  Portal  Metric  Supposition  and  Breakdown  on  Measuring  Intranet  Websites  to  Indicate  and  Improve  Effectiveness  Gains 


WHAT:  EASE  OF  NAVIGATION  (Priority  No.  5) 

WHERE:  User 

WHO:  Portal  Manager  and  Editor  Related 

WHY:  HOW: 


Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 


Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


< - BUSINESS  PROCESS - >< - BEST  PRACTICES - >  < - lEEM  Related - > 


Business  Issue: 

-  Metric  Area 

Business  Question  of 
Web  Site  Activity 

Business  Significance  of 
Web  Site  Report  Solution 

Snecific  Sub-Metric  Area 

-  Specific  Metric 

Who  Benefits  and  Why 
(User,  Portal,  Enterprise) 

lEEM  Scemcntts): 

lEEM  Constituent(s) 

Efficiency  (E) 
or 

Effectiveness(F) 

Site  Design 

Is  our  site  navigation 
effective? 

Measure  how  difficult  it  is 
for  your  visitors  to  take  the 
desired  action. 

Clicks  and  Duration  Measures: 

Usability  Study: 

User  -  Pages. 

Portal  -  Managers  learn. 

Enterprise  - 

Ed.  Proarammine. 

User  Exnerience: 

-  Navigation 

F 

Site  Design 

How  do  I  know  if  my 
site  is  optimized  for 
my  typical  web  site 
visitor? 

Track  which  browsers  and 
platforms  your  visitors  use 
so  you  can  design 
according  to  their  needs 

Web  Graphics:  Referrers,  Browsers 
used,  Search  Engines 

User  -  Pages. 

Portal  -  Managers  learn. 

Enterprise  - 

Ed.  Proarammina. 

User  Experience, 
Info.  Architecture 

-  Personalization 

F 

Site  Design 

How  can  I  make 
transactions  easier  for 
my  visitors? 

Analyze  the  seeking 
processes  to  determine 
speed  the  process  to 
download 

Click  to  Success  Rate 

User  -  Pages. 

Portal  -  Managers  learn. 

Enterprise  - 

E(F) 

Ed.  Proarammina. 

User  Exnerience: 

-  Personalization 

Site  Design 

How  do  I  evaluate  the 
success  of  design 
tradeoffs  and  choices? 

Continuously  measure  the 
effect  of  each  design 
change  on  your  visitation 

Caebin2  Statistics 

User  -  Pages. 

Portal  -  Managers  learn. 

Enterprise  - 

User  Exnerience. 

Ed.  Proarammina: 

-  Navigation 

Info.  Knowledge 

F 

Table  16,  Ease  of  Navigation  Metrics  to  Business  Requirements 
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Intranet  Portal  Metric  Supposition  and  Breakdown  on  Measuring  Intranet  Websites  to  Indicate  and  Improve  Effectiveness  Gains 


WHAT:  TOP  DOWNLOADS  /  PAGES  (Priority  No.  6) 

WHERE:  Portal 

WHO:  Portal  Manager  Related 

WHY:  HOW: 


Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 

Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


< - BUSINESS  PROCESS - >< - BEST  PRACTICES - >  < - lEEM  Related - > 


Business  Issue: 

-  Metric  Area 

Business  Question  of 
Web  Site  Activity 

Business  Significance  of 
Web  Site  Report  Solution 

Snecific  Sub-Metric  Area 

-  Specific  Metric 

Who  Benefits  and  Why 
(User,  Portal,  Enterprise) 

lEEM  Scemcnttsl: 

lEEM  Constituent(s) 

Efilciency  (E) 
or 

Effectiveness(F) 

Site  Design 

Are  there  popular 
pages  that  aren’t  easy 
to  access? 

Improve  access  to  popular 
pages  and  satisfy  your 
visitors 

Deen  Link  Correlation  tcontent  sroun 

bv  no  referreri 

User  -  Pages. 

Portal  -  Managers  learn. 

Enterprise  - 

Info  Architecture: 

-  Info  Grouping 

-  Search 

-  Content  Properties 

F 

Site  Design 

What  keywords  should 

I  use  to  maximize 
search  engine  traffic  to 
my  site? 

Maximize  traffic  to  your 
site  by  understanding  what 
terms  your  visitors  are 
using  to  locate  your  site 

(Search  engine  optimizer) 

User -Pages. 

Portal  -  Managers  learn. 

Enterprise  - 

Info  Knowledge: 

-  Content  Properties 

F(E) 

Table  17,  Top  Downloads/Pages  Metrics  to  Business  Requirements 
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Intranet  Portal  Metric  Supposition  and  Breakdown  on  Measuring  Intranet  Websites  to  Indicate  and  Improve  Effectiveness  Gains 


WHAT:  USER  SATISFACTION  SURVEYS  (Priority  No.  7) 

WHERE:  Front  End 

WHO:  Portal  Owner,  Manager,  Editor  and  User  Related 

WHY:  HOW: 


Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 

Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


< - BUSINESS  PROCESS - >< - BEST  PRACTICES - >  < - lEEM  Related - > 


Business  Issue: 

-  Metric  Area 

Business  Question  of 
Web  Site  Activity 

Business  Significance  of 
Web  Site  Report  Solution 

Snecific  Snb-Mctric  Area 

-  Specific  Metric 

Who  Benefits  and  Why 
(User,  Portal,  Enterprise) 

lEEM  Seamentfsl: 

lEEM  Constituent(s) 

Efficiency  (E) 
or 

Effectlveness(F) 

User  Experience 

Do  users  prefer  and 
use  my  site?  Why  and 
Why  not? 

Improve  satisfaction  of 
your  content  delivery  to 
help  users  do  their  jobs  and 
make  decisions  better. 

User  Sat  Survey  with  a  variety  of 
questions  related  to  the  business 
discussion 

Customer 

Portal 

Enterprise 

All  Seements 

-  All  lEEM 
Constituents 

F(E) 

Site  Design 

How  can  the  site  (UI) 
be  modified  to  best 
meet  the  demands  of 
users? 

Improve  satisfaction  of 
your  content  delivery 

User  Sat  Survey  with  a  variety  of 
questions  related  to  the  business 
discussion 

Customer 

Portal 

Enterprise 

All  Segments 

-  All  lEEM 
Constituents 

F(E) 

Table  18,  User  Satisfaction  Survey  Metrics  to  Business  Requirements 
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Intranet  Portal  Metric  Supposition  and  Breakdown  on  Measuring  Intranet  Websites  to  Indicate  and  Improve  Effectiveness  Gains 


WHAT:  Time  to  Locate  (Priority  No.  *) 

There  are  many  variables  that  eould  skew  the  results  for  time  to  reaeh  a  deeision  to  be  eonsistent  (i.e.,  individual  skill  sets  and  experienee  ean  vary 
greatly).  However,  if  improvements  are  made  in  the  other  metrie  areas  previously  listed,  they  will  eollectively  help  to  minimize  the  time  to  loeate 
desired  information  and  the  resulting  deeision  reaehed.  Nevertheless,  from  time  to  time  users  may  find  what  they  are  looking  for  sooner,  but  will 
also  continue  to  look  for  long  periods  of  time  (perhaps  as  much  as  the  approximate  50%  of  their  time  as  they  do  now)  because  they  are  finding  more 
of  what  they  are  seeking.  Regardless,  the  time  factor  is  reduced  with  respect  to  finding  what  is  sought  or  considered  desirable:  If  people  still  spend 
50%  of  their  time  looking  for  information,  they  should  have  more  pertinent  information  than  before  in  the  same  amount  of  time  -  which  should  lead 
to  better  decision  making  and  ultimately  more  effectiveness.  “Time  to  Locate”:  is  an  efficiency  metric  that  is  affected  by  increases  in  effectiveness 
elsewhere.  The  efficiency  metric  of  Time  to  Locate  in  turn  affects  effectiveness  across  the  board  because  users  will  either  have  or  can  do  more  in  less 
time.  Thus,  this  metric  area  is  an  example  of  how  effectiveness  affects  efficiency  and  then  how  this  efficiency  increase  in  turn  improves 
effectiveness. 
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Intranet  Portal  Metric  Supposition  and  Breakdown  on  Measuring  Intranet  Websites  to  Indicate  and  Improve  Effectiveness  Gains 


WHAT:  Miscellaneous 

WHERE:  Portal 

WHO:  Portal  Manager  and  Editor  Related 

WHY:  HOW: 


Business  Req,  or 
Derived  Metric 

Back  End  or 
Hard  Metric 

Front  End  or 
Soft  Metric 

Best  Bet  or 
Practice 


< - BUSINESS  PROCESS - >< - BEST  PRACTICES - >  < - lEEM  Related - > 


Business  Issue: 

-  Metric  Area 

Business  Question  of 
Web  Site  Activity 

Business  Significance  of 
Web  Site  Report  Solution 

SDCcific  Sub-Metric  Area 

-  Specific  Metric 

Who  Benefits  and  Why 

(User,  Portal,  Enterprise) 

lEEM  Seementts): 

lEEM  Constituent(s) 

Efficiency  (E) 
or 

EffectivenesstF) 

User  Experience 

How  often  do  people 
abandon  my 
registration  and  survey 
forms? 

Monitor  incomplete  survey 
and  registration  forms  and 
analyze  why  they  were 
abandoned 

Registration  abandonment  rate 

Portal 

User 

Info.  Knowledee 

E(E) 

User  Experience 

How  often  do  people 
bookmark  my  site? 

Understand  how  important 
your  content  is  to  certain 
visitors 

Bookmark  volume  trend 

Portal 

User 

I 

nfo.  Knowledge 

F 

Performance 

How  do  I  gauge  the 
performance  of  my 
caching  or  load 
balance? 

Analyze  and  understand 
what  perfomiance  benefit 
caching  has  for  my  visitors 
and  make  appropriate 
changes 

Caching  statistics 

Load  Balance  statistics 

Enterprise 

Portal 

User 

Info.  Knowledee 

E 

Web  Site 
Performance 

How  do  I  know  what 
errors  are  occurring  on 
the  web  site? 

Control  error  rates  by 
tracking  client,  server,  and 
dynamic  page/form  errors 
and  fixing  them 

Visitor  Error  Rate 

Portal 

Info.  Knowledee 

E 

Table  19, 

Miscellaneous  Metrics  to  Business  Requirements 

Portal  Manager  Questions  Related  to  lEEM  Buy-in  (Miscellaneous) 


Business  Issue 

User  Pain 

Interoperability 

Will  this  solution  work  with  my  existing  environment?  (Databases,  applications  and  hardware  platfomis) 

Data  Integration 

How  do  you  integrate  multiple  data  sets,  on  multiple  platforms  for  multiple  people? 

Productivity 

How  can  I  automate  the  process  of  distributing  the  analysis  and  reports  on  website  activity?  (and  present  them  in  a  format  that  is  understandable  and  meaningful) 

Scalability 

Can  the  system  scale  to  the  large  amount  of  Web  data  we  process  and  provide  us  with  timely  reporting? 
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APPENDIX  E:  COMMON  APPROACHES  TO  IT  ROI 


Technique 

General  Definition 

How  to  Calculate 

Advantage 

Disadvantage 

Return  on 

Investment 

(ROI) 

A  catchall  phrase  commonly 
used  for  several  ways  to  measure 
business  value  of  a  project.  ROI 
means  profit  divided  by  invest¬ 
ment,  expressed  as  a  percentage. 
As  the  numerator,  profit  can  be 
replaced  by  cost  reductions  or 
productivity  gains  derived  from 
the  operational  improvements  an 
IT  project  yields 

Revenue  or  cost  savings 
divided  by  investment 

Best  applied  to  projects 
where  all  costs  that  will 
be  incurred  or  all  cost 
reductions  that  will  be 
realized  are  known 
ahead  of  time,  usually 
from  experience  on  a 
similar  project. 

Difficult  to  apply  to 
entrepreneurial  IT 

projects  that  are 

designed  to  help  launch 
new  products,  services  or 
businesses  that  translate 
to  new  sources  of 
revenue  and  profits.  ROI 
doesn't  consider  risk, 
flexibility  &  intangibles. 

Net  Present 
Value  (NPV) 

Refers  to  the  future  net  cash 
flow  a  project  is  expected  to 
deliver,  minus  the  investment.  It 
defines  the  value  of  a  project  in 
"today's  dollars."  The  calculation 
is  based  on  the  company's  cost 
of  capital  used  for  assessing 
proposal  alternatives.  It  returns  a 
nominal  amount. 

Cash  inflow  minus  cash 
outflows  calculated  in  today's 
dollars. 

Includes  all  cash  flow 
related  to  a  project. 
Considers  the  time  value 
of  money,  or  the 
difference  in  the  value  of 
a  dollar  today  and  what 
it  might  be  three  years 
from  now. 

The  highest  NPV  doesn't 
always  correspond  to  the 
most  efficient  use  of  a 
company's  capital. 

Internal  Rate 
of  Return 

(IRR) 

One  of  several  metrics  that 
considers  the  time  value  of 
money,  IRR  expresses  the  dollar 
returns  expected  from  a  project 
as  an  interest  rate.  Once  the  rate 
is  established,  it  can  be 
compared  to  rates  earned  by 
investing  in  other  projects.  More 
informally,  IRR  is  also  known  as 
the  "hurdle  rate"  because  it's 
usually  the  lowest  rate  of  return 
that  management  will  accept. 
Typically,  a  project  must  earn  an 
IRR  that  is  several  percentage 
points  higher  than  the  cost  of 
borrowing,  to  compensate  the 
company  for  its  risk  exposure 
and  time. 

C=  all  costs  associated  with 
the  project  and  call  it . 
R=estimate  of  all  returns 
resulting  from  the  project. 

T=  how  many  months  or  years 
company  will  realize  returns. 
i=  firm's  minimum  acceptable 
rate  of  return 

Calculate  the  interest  rate: 

C=R  X  T  (i). 

Reference  a  NPV  chart  listing 
the  value  of  a  $1  annuity  and 
find  the  corresponding  interest 
rate.  Compare  that  interest 
rate  to  the  minimum 
acceptable  rate  and  determine 
if  project  will  leap  over  hurdle 
rate. 

Includes  all  cash  flow 
related  to  a  project. 
Considers  the  time  value 
of  money.  It  enables  the 
comparison  of  rates  of 
return  on  alternative 
investment  options. 

Given  two  investment 
alternatives  and 

assuming  that  both  fit 
strategic  objectives  of 
the  organization,  the 
investment  with  higher 
internal  rate  of  return 
should  be  selected. 
Conceptually  it  is  the 
easiest  method  to 

understand. 

Disadvantages:  Assumes 
cash  flows  are  reinvested 
at  the  IRR.  Cumbersome 
to  calculate  interest  rate 
when  cash  flows  vary 
widely  year  to  year. 
There  is  no  specific 
formula  that  can  be  used 
to  calculate  the  IRR;  it  is 
found  by  interpolation. 

Payback 

Period 

How  long  it  will  take  an 
investment  to  pay  for  itself 

Initial  project  investment 
divided  by  cash  inflows  (or 
cost  reductions)  per  year. 

It's  simple  and 

understandable. 

Time  value  of  money 
and  cash  or  other 
benefits  received  after 
payback  period  are  not 
recognized,  which 

determines  profit. 

Economic 
Value  Added 
(EVA) 

Measures  a  corporation's  true 
economic  profit.  The  idea  is  to 
understand  which  business  units 
best  leverage  their  assets  to 
generate  returns  and  maximize 
shareholder  value. 

Net  operating  profit  minus  an 
appropriate  charge  for  the 
opportunity  cost  of  all  capital 
invested  in  an  org.  - 
EVA=  Net  Operating  Profit 
After  Taxes  (NOPAT) 
(Capital  X  Cost  of  Capital) 

Can  more  precisely 
define  value  in  terms 
specific  to  an  enterprise. 

Complex,  proprietary 

(expensive)  and  not 
widely  used.  Metric  is 
extraordin-arily  dependent 
on  the  size  of  a  business. 
Big  operations/  projects 
tend  to  produce  big  EVAs, 
while  small  operations/ 
projects  are  much  smaller. 

Table  20,  Common  Approaches  to  IT  ROI 


145 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


146 


APPENDIX  F:  SPREADSHEET  OF  ROI  RETURNS 


Weekly 

Weekly 

Bi-Weekly 

Bi-Weekly 

Mouthly 

Mouthly 

Quarterly 

Quarterly 

Variable  Calculated 

Best  Case 

Worst  Case 

Best  Case  Worst  Case 

Best  Case 

Worst  Case 

Best  Case 

Worst  Case 

Number  of  Procedures 

No.  procedures  required  to  perform 
activity  in  a  non-optimized  portal 

50 

50 

110 

110 

310 

310 

1200 

1200 

No.  procedures  required  to  perform 
activity  in  an  optimized  portal 

45 

48 

95 

102 

285 

300 

1100 

1150 

Number  of  Hours 

No.  hours  required  to  complete  activity  in  a 
non-optimized  portal 

30 

30 

68 

68 

125 

125 

400 

400 

No.  hours  required  to  complete  activity  in 
an  optimized  portal 

Number  of  Knowledge  Workers  that 

26 

28.5 

57 

64 

110 

116.5 

350 

375 

Conduct  same  Task 

4000 

3500 

1000 

900 

500 

450 

250 

220 

Cost  of  Activity  (direct  and  Overhead) 
Additional  Cost  (Associated  to  the  optimized 

700 

700 

1200 

1200 

2400 

2400 

15000 

15000 

portal  for  this  activity) 

200 

300 

400 

700 

800 

1100 

1000 

1400 

Indirect  Costs  for  Non-Optimized  Portal 

Indirect  cost  (time  in  minutes)  assigned 
procedures  in  non-optimized  portal 

36 

36 

37.0909 

37.09090 

24.19354 

24.19354 

20 

20 

Average  cost  assigned  non-optimized  portal 
over  period  to  complete  activity 

Indirect  Costs  for  Optimized  Portal 

14 

14 

10.90909 

10.90909 

7.74193 

7.74193 

12.5 

12.5 

Indirect  cost  (time  in  minutes)  assigned 
procedures  in  optimized  portal 

34.66666 

35.625 

36 

37.64705 

23.15789 

23.3 

19.09090 

19.56521 

Average  cost  assigned  optimized  portal 
over  time  period  to  complete  activity 

Amortized  Indirect  Costs  Optimized  Portal 

17.92592 

20.10416 

14.79876 

17.93540 

10.21754 

11.12266 

12.84090 

13.44565 

Average  cost  assigned  optimized  portal 
amortized  over  one  year  time  period 

13.57407 

13.98437 

10.76367 

11.35861 

7.64444 

7.76155 

12.15909 

12.53260 

Frequency  of  Activity  (answer  only  one) 

48 

48 

24 

24 

12 

12 

4 

4 

Procedures  Gaiued  iu  Oue  Activity  Period 
Perceutage  Procedures  Gaiued  iu  Oue 

6.923076 

2.526315 

18.333333 

6.375 

38.863636 

21.88841 

157.14285 

76.66666 

Activity  Period 

Procedures  Gaiued  iu  Oue  Year  for  this 

3.84615 

1.052631 

3.030303 

-1.477272 

4.47214 

3.83497 

4.76190 

2.22222 

Activity 

92.30769 

25.26315 

80 

-39 

166.36363 

142.66094 

228.5714 

106.6666 

Perceutage  Chauge  iu  Procedure 

Productivity  for  Activity  iu  Oue  Year  (%) 
Total  Cost  of  Procedures  for  this  Activity  iu 

3.846153 

1.052631 

3.030303 

-1.477272 

4.472140 

3.834971 

4.76190 

2.22222 

Oue  Year 

33830.76 

33915.78 

29277.192 

29543.75 

29709.090 

29980.257 

61142.857 

61493.333 

Saviugs  of  Procedures  for  Activity  iu 
Optimized  Portal  over  Oue  Year 

1061.538 

37.89473 

395.53429 

-1169.204 

378.88563 

-75.78568 

1714.28571 

-160 

Perceutage  Saviugs  of  Procedures  iu 
Optimized  Portal  over  Oue  Year 

3.140646 

0.111783 

1.354569 

-3.963405 

1.28001 

-0.253463 

2.810304 

-0.260586 

Auuual  Saviugs  to  Euterprise  for  Activity 
(if  more  than  one  employee) 

Percentage  Annual  Savings  to  Enterprise 

4246153.85 

132631.57 

395534.29 

-1052284.09 

189442.81 

-34103.55 

428571.43 

-35200 

for  ActivMr 

3.15933 

0.112781 

1.373363 

-4.05962 

1.31550 

-0.26312 

2.85695 

-0.26663839 

Additional  Similar  Activities  Accomplished 
across  Enterprise 

Percentage  Additional  Similar  Activities 

8205.128 

1842.105 

842.1052 

-344.11764 

291.86602 

213.99141 

51.94805 

20.40579 

Accomplished  across  Enterprise 

Virtual  K-Workers  Gained  for  Aetivity 

4.273504 

1.096491 

3.50877 

-1.5931372 

4.86443 

3.962804 

5.194805 

2.318840 

across  Enterprise 

Percentage  Virtual  K-Workers  Gained  for 

170.9401 

38.37719 

35.0877 

-14.338235 

24.32216 

17.83261 

12.98701 

5.10144 

Activity  across  Enterprise 

4.273504 

1.096491 

3.50877 

-1.593137 

4.86443 

3.962804 

5.194805 

2.318840 

Table  21,  Comparative  ROI 

Returns  for  Different  Activities:  One  Year 
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Weekly 

Weekly 

Bi-Weekly 

Bi-Weekly 

Monthly 

Monthly 

Quarterly 

Quarterly 

Variable  Calculated 

Best  Case 

Worst  Case 

Best  Case 

Worst  Case 

Best  Case 

Worst  Case 

Best  Case 

Worst  Case 

Number  of  Procedures 

No.  procedures  required  to  perform 
activity  in  a  non-optimized  portal 

50 

50 

110 

no 

310 

310 

1200 

1200 

No.  procedures  required  to  perform 
activity  in  an  optimized  portal 

45 

48 

95 

102 

285 

300 

1100 

1150 

Number  of  Hours 

No.  hours  required  to  complete  activity  in  a 
non-optimized  portal 

30 

30 

68 

68 

125 

125 

400 

400 

No.  hours  required  to  complete  activity  in 
an  optimized  portal 

Number  of  Knowledge  Workers  that 

26 

28.5 

57 

64 

no 

116.5 

350 

375 

Conduct  same  Task 

4000 

3500 

1000 

900 

500 

450 

250 

220 

Cost  of  Activity  (direct  and  Overhead) 
Additional  Cost  (Associated  to  the  optimized 

700 

700 

1200 

1200 

2400 

2400 

15000 

15000 

portal  for  this  activity) 

200 

300 

400 

700 

800 

1100 

1000 

1400 

Indirect  Costs  for  Non-Optimized  Portal 

Indirect  cost  (time  in  minutes)  assigned 
procedures  in  non-optimized  portal 

36 

36 

37.09090 

37.0909 

24.19354 

24.19354 

20 

20 

Average  cost  assigned  non-optimized  portal 
over  period  to  complete  activity 

Indirect  Costs  for  Optimized  Portal 

14 

14 

10.90909 

10.9090 

7.741935 

7.741935 

12.5 

12.5 

Indirect  cost  (time  in  minutes)  assigned 
procedures  in  optimized  portal 

34.66666 

35.625 

36 

37.64705 

23.15789 

23.3 

19.09090 

19.5652 

Average  cost  assigned  optimized  portal 
over  time  period  to  complete  activity 

Amortized  Indirect  Costs  for  Optimized 
Portal 

17.92592 

20.10416 

14.79876 

17.93540 

10.21754 

11.12266 

12.84090 

13.44565 

Average  cost  assigned  optimized  portal 
amortized  over  one  year  time  period 

13.52777 

13.91927 

10.67595 

11.21563 

7.52748 

7.608777 

12.04545 

12.38043 

Frequency  of  Activity  (answer  only  one 
choice  below) 

96 

96 

48 

48 

24 

24 

8 

8 

Procedures  Gained  in  One  Activity  Period 
Percentage  Procedures  Gained  in  One 

6.92307 

2.5263 

18.3333 

6.375 

38.8636 

21.8884 

157.1428 

76.6666 

Activity  Period 

Procedures  Gained  in  One  Year  for  this 

3.84615 

1.05263 

3.0303 

-1.47727 

4.47214 

3.83497 

4.761904 

2.2222 

Activity 

184.6153 

50.5263 

160 

-78 

332.7272 

285.321 

457.1428 

213.33 

Percentage  Change  in  Procedure 

Productivity  for  Activity  in  One  Year  (%) 
Total  Cost  of  Procedures  for  this  Activity  in 

3.84615 

1.05263 

3.0303 

-1.47727 

4.472140 

3.83497 

4.761904 

2.2222 

One  Year 

67430.76 

67515.78 

58077.19 

58343.75 

58509.0909 

58780.257 

121142.857 

121493.33 

Savings  of  Procedures  for  Activity  in 
Optimized  Portal  over  One  Year 

Percentage  Savings  of  Procedures  in 
Optimized  Portal  over  One  Year 

2353.846 

3.492353 

391.5789 

0.580116 

1268.261 

2.186657 

-1594.659 

-2.735264 

1666.862 

2.854216 

1028.686 

1.752446 

4571.428 

3.77804 

1173.333 

0.966501 

Annual  Savings  to  Enterprise  for  Activity 

(if  more  than  one  employee) 

9415384.62 

1370526.31  1268261.563 

-1435193.18  833431.085  462908.7637 

1142857.1 

258133.33J 

Percentage  Annual  Savings  to  Enterprise 
for  Aetivilr 

3.50274 

0.582706024  2.201827701 

-2.76846 

2.893777 

1.785837 

3.8093 

0.97772 

Additional  Similar  Activities  Accomplished 


across  Enterprise 

16410.25 

3684.210 

1684.210 

-688.2352 

583.7320 

427.9828 

103.8961 

40.8115 

Percentage  Additional  Similar  Activities 
Accomplished  across  Enterprise 

4.273504 

1.096491 

3.50877 

-1.593137 

4.864433 

3.962804 

5.194805 

2.31884 

Virtual  K-Workers  Gained  for  Activity 
across  Enterprise 

170.9401 

38.37719 

35.0877 

-14.33823 

24.32216 

17.83261 

12.98701 

5.101449 

Percentage  Virtual  K-Workers  Gained  for 
Activity  across  Enterprise 

4.273504 

1.096491 

3.50877 

-1.593137 

4.864433 

3.962804 

5.194805 

2.31884 

Table  22,  Comparative  ROI  Returns  for  Different  Activities:  Two  Years 
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A 

B 

C 

D 

E 

F  G 

H 

J 

K 

Variable  Calculated 

Best  Case 

Worst  Case 

DiflToronce  ; 

Number  of  Procedures 

3 

No.  pioceduies  required  to  perfonn  edivityin  anon-optimixed  portal 

50 

50 

No.  procedures  requited  to  perform  activity  in  «n  optiissedportal 

45 

48 

-3 

S  INiutiber  of  Hours 

No.  hours  required  to  complete  activity  in  a  non- optimized  portal 

30 

losood* 

30 

108000* 

7 

No.  hours  required  to  complete  activity  in  an  optimized  portal 

26 

93600* 

28.5 

102600 

-2.5 

Nuitdier  of  Knowledge  Workers  that  Conduct  same  Task 

4000 

3500 

-500 

Cost  of  Activity  (direct  and  Overhead) 

700 

700 

10 

Additional  CostfAssociatedto  the  onttmizedDoital  for  this  activitu’i 

200 

300 

-100 

11 

bidireci  Costs  for  Non-Optiitiized  Portal 

12 

Indirect  cost  (time  in  minutes)  assigned  procedures  in  non-optimizedpoital 

3^ 

2160* 

0.0CCI462963' 

36' 

2160' 

0.000462963' 

13 

Average  cost  assigned  non-optimizedportal  over  period  to  complete  activity 

14 

0.07142857? 

14 

0.071428571' 

14 

Indirect  Costs  for  Optimized  Portal 

IS 

Indirect  cost  (time  in  minutes)  assigned  procedures  in  optimized  portal 

34.66666667' 

2080' 

0.000480769' 

35.62? 

2137.? 

0000467836' 

-57.5 

16 

Average  cost  assigned  optimized  portal  over  time  period  to  complete  activity 

17.92592593' 

0.055785124 

m.666666f 

606.666666? 

0.86666666?  20.1041666? 

0.0^740933' 

96? 

66? 

0.9? 

-2.178240741 

17 

Amortized  Indirect  Costs  for  Optimized  Portal 

IS 

Average  cost  assigned  optimizedportal  amortized  over  one  year  time  period 

13.57407407' 

0.0736698? 

610.8333333' 

4.166666667' 

1398437? 

0.07150838' 

6712? 

6.2? 

0  410301926 

19 

Frequency  of  Activity  (answer  only  one  choice  beloi^ 

48* 

48' 

20 

Weekly  (enter  the  value  of  1  if  this  applies) 

1 

48* 

1 

48' 

21 

Bi-Weekly  (enter  the  value  of  1  if  this  applies) 

o' 

o' 

22 

Monthly  (enter  the  value  of  1  if  this  qiplies) 

o' 

o' 

23 

Bi-Monthly  (enter  the  value  of  1  if  this  ^plies) 

d* 

o' 

24 

Quarterly  (enter  the  value  of  1  if  this  ^plies) 

o' 

o' 

25 

Semi-Annually(entetthe  value  of  1  if  this  ^hes) 

o' 

o' 

26 

AmuallT  ('enter  the  value  of  1  if  this  anoHes'i 

o' 

o' 

28 

Procedures  Gained  in  One  Activity  Period 

6.923076923' 

5192307692* 

2.526315789' 

50.52631579* 

-4,396761134 

29 

Percentage  Procedures  Gained  in  One  Activity  Period 

3.846153841? 

0038461538' 

1.038461538' 

1.052631579' 

0010526316' 

1010526316' 

-2.793522267 

30 

Procedures  Gained  in  One  Year  for  this  Activity 

92.30769231  2400 

2492.307692' 

25.26315789' 

'  24CD' 

2425.263158' 

' -67.04453441 

31 

Percentage  Change  in  Procedure  Productivity  for  Activity  in  One  Year  CHi) 

3.84615384^ 

0.038461538' 

1.038461538' 

1.052631579' 

001052631? 

1.01052631? 

-2.793522267 

32 

Total  Cost  ofProcedures  for  this  Activity  in  One  Year 

33830.76923' 

2492.307692' 

33915.7894? 

2425.263158' 

85.02024291 

d4 

35 

Savings  ofProcedures  for  Activity  in  Optimized  Portal  over  One  Year 

1061.538462 

233.7692308 

3^00 

1292.307692 

37.89473684 

315.789473? 

33600 

353.684210? 

-1061.538462 

36 

Percentage  Savings  ofProcedures  in  Optimized  Portal  over  One  Year 

3.140646336' 

0.031406463' 

33800' 

0.111783884 

0.001117839' 

33900' 

-3.028862451 

37 

Annual  Savings  to  Enteiprise  for  Activity  (if  more  than  one  employee) 

4246!53i' 

'1326315789' 

-4113522267 

;-lH 

Percentage  Annual  Savings  to  Ejdeiprise  for  Activi^ 

3.159336' 

003159336' 

134400200 

134400000 

0.11278166? 

0.00112781? 

117600330 

117600000 

-3.046554291 

40 

Additional  Similar  Activities  Acconqilished  across  Enterprise 

8205.12820? 

369230.7692' 

1842.105263' 

88421.05263' 

-6363.02SI42 

41 

42 

Percentage  Additional  Similar  Activities  Acconqilished  across  Enteiprise 
Virtual  K- Workers  Gained  for  Activity  across  Enteiprise 

'’170.9401709' 

'  38.37719298' 

-132,562978 

43 

Percentage  Wrtual  K- Workers  Gained  for  Activity  across  Enteiprise 

4.273504274 

0.042735043' 

1.0427350^' 

4170.940171’ 

1.096491228' 

0.01096^12' 

1.01096^12' 

3538377193' 

-3.177013)45 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 

N  •< 

►  N  N  Weekly  /  B-Weekv  /  Monthly  /  Quarterly  /  Comparattve  view  ).  Expanded  View  (Weekly)  i  Comments  sheet  / 

1*1 

J  >ir 

Reddy 


Table  23,  Expanded  Field  view  of  Comparative  ROI  Weekly  Returns 
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^  Microsoft  Excel  -  ABCRange  R0l_03 


Si  R!e  Edit  View  Insert  Format  Toc4s  Data  Window  Help 

iOl-^lOO^o  -  l?l  TimesNewftoman 

j  ts  sj  .  a  «  oi 


/  U  ES3g  $% 


Grairt  A.  Jacoby: 

Tot^  Number  oF 
Procedures  divided  by 
Total  Cost;  (B  or  6  3)  /  (0 
G9) 


Grant  A.  Jacoby: 

Number  of  Procedures 
per  Seconds  For  work 
requr^:  (6 or  G4) /  (Bo 
G  6)  X  3600  seconds 


Grant  A.  Jacoby: 

Ratio  oF  Procedures 
gained  in  optimized  portd 
non-optimized 
portal  during  one  year 
expressed  as  a 
percentage:  (Cor  H  31) 
:x  100 


Grant  A.  Jacoby: 

Total  Cost  oF  Total 
Number  of  Procedures 
Gained  From  the 
optimized  portal  during 
the  frequency  of  the 
Activity  being  completed 
over  on  year,  I.e.,  total 
procdures  gained  x 
amotized  cost  of  each 
procdure:  (C  or  H  32)  x 
(B  or  G  17) 


10  Additional  Cast  (Associated  to  the  optoizedportal  for  this  acUvity) 

11  IndirectCostsibrNoii-OptiinuedPoital 

12  Indiiect  cost  (time  inmmutes)  assignedproceduresinnoivoptinuzedpoftal 

13  Average  cost  assigned  non-optimized  portal  over  period  to  complete  activity 

14  Indirect  Costs  for  Optimized  Portal 

15  Indirect cost(time inminutes)  assignedproceduresinoptimizedportal 

16  Average  cost  assigned  optimized  portal  overtime  period  to  complete  activity 

17  Amortized  Indirect  Costs  for  Optimized  Portal 

18  Average  cost  assigned  optimized  portal  amortized 

19  FrequencyofAEtiiity(answer  only  one  choice  below) 

20  Weekly  (enter  the  value  ofl  if  this  applies) 

21  Bi-Weekly(eclerthe  value  ofl  if  this  applies) 

22  Monthly  (enterihe  value  of  1  if  this  ^plies) 

23  Bi-Monthly  (enter  the  value  ofl  if  this  ^plies) 

24  Quarterly  (enterthe  value  of  1  ifthis  applies) 

25  Semi-Annually(entetihe  value  ofl  ifthis  applies) 

Annually  (enter  the  value  ofl  ifthis  applies) 

28  Procedures  Gained  in  One  Activity  Period 

29  Percentage  Procedures  Gained  in  One  Activity  Period 
^  Procedures  Gained  in  One  Year  for  this  Activi^ 

31  Percentage  Change  in  Procedure  Productivity  for  Activity  in  One  Year  (Aii) 

32  Total  Cost  of  Procedures  for  this  Activity  in  One  Year 

J4 

%  Savings  of  Procedures  for  Activity  in  Optimized  Portal  over  One 

36  Percentage  Savings  of  Procedures  in  Optimized  Portal  overOtie  Year 

37  Annual  Savings  to  Entetprise  for  Activity  (if  more  thangdo  employe  a) 

38  JenwUHe  Annual 

jy 

40  Additional  Similar  Activities  Acconqilishie^,4cross  Qiteipris^ 

41  Percentage  Additional  Similar  Activitiei^coit^lished : 

42  Virtual  K- Workers  Gained  for  A^tlnty  across  Entetpri) 

43  Percentage  ^Trtual K-Workpt^Gained for  Activity  ^Cn)gg^Ente|jtrisc 

44  _ 

Grant  A.  Jacoby: 

Increase  of  ratio  of 
procedurs  gained  within 

47  an  optimized  portal 

48  duinrg  one  period 
expressed  as  a  percent!: 

(CorH29)xlOO 


Grant  A.  Jacoby: 

Difference  In  Procedures 
gained  between  Activity 
Frequency  during  one 
year  within  optlrrized  and 
non-cptimized  portals;  (C 
or  H  30)  -  (D  or  1 30) 


Grant  A.  Jacoby: 

Ratio  of  Hours  to 
complete  an  activity 
between  optimized  and 
rion-optirrized  portal:  (B 
orG7)/(B  orG6) 


&ant  A.  Jacoby: 

Total  Amortized  Cost  c^ 
Total  Costs,  i.e.,  Cost  of 
Activity  and  Additional 
Metrics  Cost;  (E  or  J  16) 
+  (E  or  J  18) 


Grant  A.  Jacoby: 

Addition^  Cost 
associated  to  metrics 
analysis  and  application 
Amortized  over  the 
frequency  of  activity  in 
eyear;  (BorG  1CI)/(E 
JIB) 


Grant  A.  Jacoby: 

Total  Cost  showing  what 
difference  in  Procedures 
gained  between  Activity 
frequency  during  one 
year  within  optimized  and 
-cvtimized  portals  if 
those  same  procedures 
gained  had  been  charged 
averse  direct  cost  of  e 
in-optimized  portal:  (B 
G  30)  /  (B  or  G  13) 


Grant  A.  Jacoby: 

Total  Number  of 
Procedures  Gained  from 
the  optimized  portal 
dirlng  the  Frequency  of 
the  Activity  being 
completed  over  on  year; 
(C  orH28)x(Bor619) 


Grant  A.  Jacoby: 

ROl  of  total  number  of 
knowledge  workers  (or 
groups)  that  complete 
tNs  activity  as  part  of 
;ir  job,  plus  the 
addtional  knowledge 
workers  gained  from 
completing  the  Activity 
through  an  opOmlzed 
portal  DIVIDED  by  the 
Total  number  of 
knowledge  workers  (or 
groups)  that  com$ilete 
tNs  activty  as  part  of 
their  job,  WIThKXiT  the 
additional  knowledge 
"  workers  gained  from 
completing  the  Activity 
through  an  optimized 
portal  expressed  as  a 
percentage;  (C  or  H  43) 
xlOO 


GraiA  A.  Jacoby: 

Total  number  of 
knowledge  workers  (or 
groups)  that  complete 
this  activity  as  part  c# 
their  job,  plus  the 
additicnal  knowledge 
workers  gained  from 
completrig  the  Activity 
through  an  optimized 
portal  DIVIDED  by  the 
Total  number  of 
knowledge  workers  (or 
groups)  that  complete 
this  activity  as  part  of 
their  job,  WITHOUT  the 
additional  knowledge 
workers  gained  from 
completing  the  Activity 
through  an  optimized 
portal  expressed  as  a 
decimal;  (D  or  1 43)  -1 


Grant  A.  Jacoby: 

AH  Costs  incurred  durir:g 
the  frequency  of  the 
Activity  over  one  year, 
I.e.,  direct,  overhead 
and  associated:  (D  or  I 
35)  -(■  (B  or  G  10) 


Grant  A.  Jacoby: 

Total  number  of 
knowledge  workers  (cr 
groups)  that  complete 
this  activity  as  part  of 
their  job,  plus  the 
additional  knowledge 
workers  gained  from 
completing  the  Activity 
through  an  optimized 
portal  DIVIDED  by  the 
Total  number  of 
knowledge  workers  (or 
groups)  that  complete 
this  activity  as  part  of 
their  job,  WITHOUT  the 
additional  knowledge 
workers  gairied  frixri 
coirpleting  the  Activity 
through  an  optimized 
portal;  (E  orJ43}/(B  or 


Grant  A.  Jacoby: 

Total  number  of 
knowled^  workers  (or 
groups)  that  complete 
this  activity  as  part  of 
their  job,  plus  the 
additional  knowledge 
workers  gained  from 
completing  the  Activity 
through  an  optimized 
portal:  (B  or  G  8)  (B  or 
G42) 


Grant  A.  Jacoby: 

Ratio  of  Acbvitles  gained 
from  an  optimized  portal 
over  a  one  year  period 
by  all  knowledge  workers 
or  groins  v^o  do  this 
activity  over  the  same 
variaNes  with  a  non- 
optimized  portal:  (E  or  J 
41) /(Fork  41) 


Graitt  A.  Jacoby: 

Total  Activities  from  noi 
optimized  portal  by 
frequency  of  activity 
applied  across  knowtedge 
workers  or  groups 
coraducting  this  activity 
over  a  one  year  period, 
plus  the  activities  gained 
If  the  same  variables  are 
applied  to  an  opUrrHzed 
portal;  (F  or  K  41)  -I-  (B  oi 
G40) 


Grant  A.  Jacoby: 

Total  Activities  from  noo- 
optlmized  portal  by 
frequency  of  activity 
applied  across  knowledge 
workers  or  groups 
conducting  this  activity 
over  a  one  year  period; 
(BorG  19)  x(BorG8) 


Grant  A.  Jacoby: 

Ratio  of  Activities  gained 
From  an  optimized  portal 
a  one  year  period 
by  all  knowledge  workers 
groups  who  do  this 
activity  over  the  same 
variables  with  a  non- 
optimized  portal 
expressed  as  a  decimal: 
(0  or  1 41)  -1 _ 
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Table  24,  Field  Comments  Explaining  ROI  Calculations 
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APPENDIX  G:  GLOSSARY  _ 

- Quantitative  - Qualitative  -  Both 

Accessibility:  Information  is  considered  accessible  when  it  is  available,  reachable,  and 
understandable.  (Assuming  the  information  exists,  ideally  availability  could  occur  from 
any  point  of  access  of  and  point  of  entry  to  the  information  system.  Although  how 
reachable  information  is  depends  on  a  number  of  factors,  in  this  context  the  more 
information  is  compatible  with  a  shared  Information  Architecture,  the  easier  it  is  to 
retrieve.  How  this  information  is  then  packaged  and  presented  in  a  fashion  that  can  be 
understood  by  the  knowledge  worker  is  the  last  stage  of  accessibility.) 

(Note:  Accessibility  also  has  another  meaning  in  terms  of  access  or  permissions  to  get 
information.) 

Additive:  The  term  that  is  characterized  by  the  addition  of  independent  statistical  sums 
that  are  representative  of  the  sum  of  their  parts  when  summed  as  a  whole.  For  example, 
when  the  total  page  views  for  each  page  in  a  yrppt,  such  as  /office,  are  summed  by  simple 
addition  that  sum  correctly  represents  the  total  page  views  for  that  vroot.  Antonym  non¬ 
additive 

Associative  relationship:  A  relationship  between  items  that  are  closely  related 
conceptually  but  not  hierarchically  and  are  not  members  of  an  equivalence  set. 

Attempted  Downloads:  The  number  of  attempted  downloads  as  indicated  by  the  HTTP 
Status  for  acknowledged  page  request.  See  also  Successful  Downloads. 

Average  minutes  per  usage  day:  The  average  number  of  minutes  spent  on  the  website, 
category,  channel,  or  application  during  the  day,  per  visiting  person. 

Average  minutes  per  usage  month:  The  average  total  number  of  minutes  spent  on  the 
website,  category,  channel,  or  application  during  the  month,  per  visiting  person. 

Average  minutes  spent  per  unique  page:  The  average  number  of  minutes  spent  on  each 
unique  page  during  the  day. 

Average  unique  pages  per  visitor  in  a  month:  The  average  number  of  different  pages 
viewed  per  day  over  the  course  of  the  month  by  those  persons  visiting  the  domain,  global 
domain,  property,  or  category.  A  unique  page  is  defined  as  a  specific  URL  that  was 
successfully  loaded  by  the  browser  at  least  once  in  the  day.  Even  if  the  page  was  viewed 
many  times,  the  page  is  counted  only  once  under  the  unique  page  definition.  In  the  case 
of  non-Web  content,  unique  pages  are  the  equivalent  of  unique  window  titles,  if 
applicable,  or  if  not,  are  undefined  and  set  to  zero.  The  multiple  URLs  displayed  within  a 
framed  page  are  not  counted  as  additional  unique  pages;  rather,  all  credit  is  given 
exclusively  to  the  host  URL,  which  does  not  change  as  users  update  constituents  of  the 
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page.  Therefore,  websites  utilizing  frames  may  show  lower  unique  page  estimates  than 
websites  that  do  not  utilize  frames. 

Average  usage  days  per  visitor  per  month  (Frequency):  The  average  number  of  different 
days  in  the  month,  per  person,  in  which  a  website,  category,  channel,  or  application  was 
visited.  A  person  is  defined  as  a  visitor  if  they  access  at  least  one  "page"  of  content  within 
the  website,  category,  channel,  or  application.  A  day  is  defined  as  12:00  midnight 
through  11:59:59  PM. 

Authoritative  Flags:  An  indication  that  the  information  presented  has  gone  through  a 
screening  process  to  discern  its  credibility,  i.e.,  confirmation  that  information  is  timely 
and  trustworthy. 

Best  Bets:  Results  from  an  authoritative  Intranet  sponsored  site  maintained  by  an  internal 
division  or  group  that  is  regularly  updated  and  contains  large  amounts  of  information 
relevant  to  the  search  topic  and  does  not  require  any  special  security  permission  to  view 
it. 

Best  Practices:  Effective  means  of  doing  work  in  a  preferred  way  that  takes  place  and 
changes  over  a  period  of  time  based  originally  from  guidelines. 

Clickstream  Analysis:  A  Web  analytics  software  that  tracks  and  measures  all  visitor 
behavior  (mouse  movement  and  links  invoked)  online  and  offline  both  quantitatively  and 
qualitatively  to  assess  user  behavior  patterns. 

Cluster:  A  group  of  servers.  Specifically,  a  cluster  is  a  group  of  servers  assigned  to  one 
sub-domain  of  an  enterprise.  Synonym:  Domain. 

Cognitive  Factor:  The  critical  analysis  ability  of  an  individual.  For  example,  in  Web 
analytics  should  attempt  to  account  for  the  cognitive  ability  of  the  knowledge  worker  to 
further  link  associated  information  and  to  take  into  account  additional  commonalities  and 
solutions  that  are  not  readily  apparent  or  realizable  by  quantitative  measurements  alone. 

Communications:  Human  and  technological  systems  used  for  sending  and  receiving 
messages.  In  the  case  of  exercising  Web  analytics,  communications  is  what  supports  the 
engagement  of  the  knowledge  worker. 

Communication  of  Authoritativeness  and  Importance:  Communicating  to  the  knowledge 
worker  the  credibility  of  an  information  system  within  that  system  to  ensure  confidence 
and  trust.  See:  Authoritative  Flags. 

Communication  of  Understanding:  Communicating  to  the  knowledge  worker  meaning 
and  significance  to  ensure  acceptance  and  engagement,  which  further  increases  the 
audience’s  propensity  to  act  on  information  in  a  common  way,  i.e.,  a  Glossary  of  Terms 
and  Definitions,  Mouse-over,  and  best  practices. 
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Community:  Enterprise  communities  are  based  on  groups  of  shared  interest.  A 
community  can  be  a  group  or  a  task,  but  they  both  share  a  common  interest. 

Composition:  The  percentage  of  a  website,  category,  channel,  or  application's  visitors 
that  belong  to  a  specific  demographic  grouping. 

Content  and  Content  Properties:  Content  -  All  documents  —  help  text,  search  results, 
forms,  or  application  information  —  that  is  delivered  to  an  knowledge  worker  via  a 
website,  or  application.  It  is  both  subject  and  specific  information.  For  example, 
“People”  is  a  subject  and  a  person  by  name  is  specific  information.  Content  is  organized 
and  found  depending  on  its  properties  and  associations 

Content  Properties  -  How  content  is  described  in  order  to  affect  the  associations  made  in 
information  retrieval  and  presentation.  The  characteristics  of  a  content  item  make  up  its 
properties,  such  properties  (a.k.a.  attributes)  include  author,  length,  name,  etc. 

Contextualization:  The  process  of  placing  an  idea,  information  item  or  knowledge  worker 
in  a  relevant  environment  to  enhance  understanding.  See:  Personalization  and 
Customization. 

Customization:  Ability  for  knowledge  workers  to  self-configure  the  contents/constituents 
of  their  domain  site.  In  essence,  individuals  are  allowed  to  subscribe  to  constituents,  or 
sub-  constituents  of  a  portal,  including  navigation,  content,  search,  etc.,  in  order  to 
provide  and  store  information  (i.e.,  like  the  notion  of  “my  saved  searches”)  most  relevant 
to  their  interests,  position  or  role. 

Data:  Information,  often  in  the  form  of  facts  or  figures  (for  example,  numbers,  text, 
images,  and  sounds)  obtained  from  experiments  or  surveys,  in  a  form  that  is  suitable  for 
storage  in  or  processing  by  a  computer  used  as  a  basis  for  making  calculations  or  drawing 
conclusions. 

Data  Mining:  Analysis  of  data  in  a  database  using  tools  which  look  for  trends  or 
anomalies  without  knowledge  of  the  meaning  of  the  data. 

Data  Warehouse:  For  any  Web  analytics,  the  system  for  storing  Web  analytic  data  for  the 
enterprise  Web  servers. 

“Discoverability”:  -  Facilitate  information  discovery  through  presentation  of  relative 
associations  which  lead  to  better  decisions,  increased  productivity  and  effectiveness  that 
would  not  have  been  likely  otherwise.  Discoverability  requires  three  events  to  happen:  1) 
associative  connections  between  content  items  are  created;  2)  the  information/  knowledge 
is  presented  to  illustrate  relative  associations  that  can  be  acted  upon;  and  3)  the 
knowledge  worker  has  the  cognitive  ability/recognizes  there  is  an  action  to  be  taken. 

Digital  media  reach  percentage:  The  percentage  of  projected  individuals  who  visited  a 
specific  website,  category,  channel,  or  application  among  the  total  number  of  projected 
individuals  using  any  digital  media  during  the  course  of  the  reporting  period. 
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Domain:  This  term  is  used  to  identify  the  domain  of  an  enterprise.  See  also:  Sub-domain. 


Domain  Integration  Framework:  Representation  of  the  relationships/strueture  between 
the  key  elements  within  a  design.  It  shows  how  these  elements  interact  and  must 
“transfer”  information,  so  that  the  architect  can  build  the  environment.  Examples  are 
information  maps  (document  elements),  server  topography,  scope  of  services, 
permissions  (ownership). 

Download:  Any  file  that  is  sent  to  a  client  by  an  HTTP  server  with  selected  extension, 
such  as  .exe,  .doc,  .zip,  etc. 

Drilling:  To  analyze  data  in  more  depth  by  moving  down  or  up  levels  within  a  category 
or  classification,  to  include  date  hierarchies.  An  example  would  be  analyzing  page  views 
for  June  2000,  then  "drilling-down”  to  June  1,  2000. 

Editorial  Programming:  The  editor’s  experience  should  be  a  straightforward  and 
consistent  use  of  accepted  best  practices  and  best  tools  available  (from  within  the 
company)  to  organize,  access  and  re-use  information  in  a  desired  manner  to  produce  a 
desired  outcome. 

Engagement:  The  more  utility  a  system  provides  the  more  recognized  and  compelling  it 
becomes.  In  other  words,  there  is  a  gradual  synergy  that  manifests  from  enthusiastic 
knowledge  workers  through  escalating  levels  of  commitment  up  to  outright  ownership  as 
a  result  of  perceived  usefulness  of  a  system/service. 

Equivalence  relationship:  A  relationship  between  items  that  is  conceptually  equivalent. 
Ex:  ASP  =  Active  Server  Pages 

External  Referrer:  In  contrast  to  internal  referrer,  external  referrers  are  the  websites 
which  are  not  managed  by  the  enterprise,  and  are  therefore  generally  not  tracked. 
Examples  are  Yahoo,  Lycos,  Excite,  etc  or  a  business  partner  domain. 

Extranet:  An  Extranet  is  somewhat  similar  to  an  Intranet.  Extranets  are  designed 
specifically  to  give  external  limited  access  to  certain  files  using  the  Internet  protocol  and 
the  public  telecommunication  system  to  securely  share  part  of  a  business's  information  or 
operations  with  suppliers,  vendors,  partners,  customers,  or  other  businesses. 

Finding:  This  is  a  process  employed  by  knowledge  workers  to  seek  the  information  they 
need  to  do  their  jobs. 

QUID:  QUID  stands  for  Global  Unique  Identifier,  a  unique  32-character  hexadecimal 
string  which  is  generated  for  each  user  who  accepts  cookies. 

GUIDed  User:  A  user  assigned  a  GUID  in  a  cookie.  Also  referred  to  as  GUIDs  and 
users. 
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Guideline:  Guidance  on  how  to  meet  established  standards.  Over  time  guidelines 
become  best  practices. 

Hierarchical  relationship:  A  relationship  between  items  based  on  one  item  being  a  part  of 
another,  one  item  being  an  instance  of  another,  or  one  item  being  descended  from 
another.  This  creates  linkages  from  broader  to  narrower,  generic  to  specific  or  parent  to 
child. 

Hit:  In  Web  activity,  a  hit  is  registered  when  a  file  is  sent  to  a  client  by  an  HTTP  server. 
This  includes  graphics,  HTML,  EXEs,  etc. 

HTTP  Status  Codes:  The  code  associated  with  a  page  request  that  is  recorded  in  the  IIS 
log  for  each  HTTP  method  exchange  between  client  and  server.  Eor  example,  a 
successful  page  request  results  in  an  HTTP  code  200  or  OK  and  can  be  imported  into  a 
log  file  for  the  data  warehouse.  Occasionally  there  is  a  404  page  error  which  occurs 
when  a  server  has  not  found  anything  matching  the  request  URL  (no  indication  is  given 
on  duration  of  condition). 

Index:  The  composition  of  the  website,  category,  channel,  or  application  compared  to  the 
demographic  composition  of  the  universe. 

Information:  The  meaning  of  data  as  it  is  intended  to  be  interpreted  by  people.  Data 
consists  of  facts,  which  become  information  when  they  are  seen  in  context  and  convey 
meaning  to  people. 

Information  Architecture:  The  information  architecture  (lA)  of  a  corporation  is  the  sum 
and  organization  of  all  its  data,  taxonomies,  tools  and  products.  lA  for  Web  analytics 
needs  to  encompass  not  all  the  lA  that  is  available  but  only  the  best  elements  of  these 
groups  which  can  be  further  developed  and  integrated  to  improve  control  of  content  and 
context  to  meet  knowledge  workers  needs  and  to  exceed  their  expectations  as  well. 

Internet:  The  Internet  is  a  public  cooperative  of  networks  and  gateways  around  the  world 
that  uses  a  portion  of  the  total  resources  of  the  currently  existing  public 
telecommunication  networks  that  use  of  a  set  of  protocols  called  TCP/IP  (for 
Transmission  Control  Protocol/Intemet  Protocol.  Two  recent  adaptations  of  Internet 
technology,  the  intranet  and  the  extranet,  also  make  use  of  the  TCP/IP  protocol. 


Intranet:  An  Intranet  is  a  private,  secured  information  portal  designed  specifically  for  the 
internal  communications  of  small,  medium  or  large  businesses,  enterprises,  governments, 
industries  or  financial  institutions  of  any  size  or  complexity.  Intranets  can  be  custom- 
designed  to  fit  the  exact  needs  of  businesses  no  matter  where  they  are  situated.  Users, 
also  referred  to  as  knowledge  workers,  of  Intranets  consists  mainly  of  company 
employees  and  business  partners. 
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Internet  Architecture:  All  constituents  used  to  provide  a  means  of  access  or  transfer  to  a 
point  of  entry. 

Information  Grouping  &  Segmenting:  Information  grouping  is  the  logical  collection  of 
similar  and  relevant  information,  such  as  Content  Nuggets  and  Categories  respectively. 
Another  aspect  of  this  is  information  segmenting  in  which  only  the  relevant  parts  of  a 
document,  i.e.  a  paragraph  or  two  are  extracted  and  placed  into  fragmented  groupings 
instead  of  the  entire  document.  For  example,  first  200  words  of  a  document  returned  as 
part  of  a  search  result. 

Information  Life  Cycle:  The  events  often  repeated  again  and  again  in  maintaining  the 
relevance  and  accessibility  of  content  in  an  information  system.  These  events  include 
updating,  versioning,  archiving  and,  when  necessary,  deleting.  See:  Retention 

Information  System:  A  procedure  that  combines  and  organizes  related  elements  into  a 
complex  whole  to  achieve  meaningful  representation  of  data. 

Internal  Referrer:  In  contrast  to  external  referrer,  internal  referrers  are  the  websites 
managed  within  an  enterprise  Intranet. 

No  Referrer  Includes  the  following  ways  a  user  can  enter  a  website: 

•  favorites  or  bookmarks 

•  a  manually  typed  URL 

•  embedded  links  in  documents  or  e-mail  messages 

•  default  start  page  of  a  browser 

•  use  of  a  browser  configured  not  to  pass  headers 

•  server  site  redirect 

Knowledge:  Understanding  gained  through  experience  or  study 

Knowledge  Worker:  One  who  gathers,  analyzes,  adds  value  and  communicates 
information  to  empower  decision-making.  (The  nature  of  ‘k-work’  is  ad  hoc,  demand- 
driven  and  creative,  both  in  the  ability  to  create  new  knowledge  greater  than  the  sum  of 
its  parts  and  in  the  ability  to  present  the  knowledge  in  a  highly  communicative  way). 

Log  Availability:  An  indicator  as  to  the  whether  log  files  are  present  and  ready  for 
importation  by  analytics  team  to  be  aggregated  into  the  data  warehouse.  Factor  affecting 
log  availability  includes  network  anomalies,  server  outages,  and  corrupt  disks. 

Metadata:  Information  about  a  content  item  derived  automatically  or  by  human  analysis. 
Types  of  metadata  include:  administrative  (managing  and  administering  content), 
descriptive  (describing/identifying)  and  technical  (related  to  how  a  system  functions  of 
how  metadata  behaves).  Metadata  can  be  expressed  in  name/value  pairs.  See:  Surrogate 
Ex:  author  :  Jane  Doe 

UID  :  7395ZX32Y2001 
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Metadata  Schema:  Accumulation  of  metadata  representing  classes  of  information 
common  to  all  members  of  a  given  content  sent.  Metadata  schema  is  part  of  an  overall 
schema.  See;  Schema 

Metrics:  Though  primarily  numeric,  mainly  quantitative  and  some  qualitative  facts  that 
are  associated  with  Web  usage,  such  as  the  count  of  unique  users  for  a  specific  page  on  a 
given  day. 

Minutes  per  Page  View:  Total  minutes  visitors  spent  on  the  sites  divided  by  total  page 
views. 

Mean  Time  Between  Failure  (MTBF):  MTBF  measures  how  long  a  server  runs  before  it 
"fails"  or  needs  to  be  rebooted. 

Navigation:  Method  of  moving  through  the  domain  framework  by  way  visual 

presentation  and  consistent  choices.  Navigation  has  2  basic  types;  local  (also  known  as 
vertical)  and  global  (also  known  as  horizontal). 

Local  navisation  presents  choices  leading  to  subtopics  or  sub-areas  of  a  site,  usually 
defined  by  one  of  its  main  menu  subjects  (often  referred  to  as  a  “drill  down”). 

Global  navisation  presents  choices  leading  to  other  main  areas  of  a  site,  i.e.  Home; 
Search;  About.  Global  navigation  is  consistent  throughout  an  information  system  which 
allows  knowledge  workers  to  go  across  portals  with  a  sense  of  familiarity. 

Non-additive:  The  term  that  refers  to  not  having  a  numerical  value  that  is  equal  to  the 
sum  of  the  parts.  In  Web  analytics,  this  term  is  used  in  application  of  the  summation  of 
unique  users  and  unique  pages. 

Null  User:  A  user  whose  browser  cannot  accept  a  cookie,  or  a  user  who  has  chosen  not 
to  accept  a  cookie,  or  a  user  who  has  not  been  to  a  site  on  which  cookies  are  issued.  We 
cannot  assign  a  permanent  QUID  to  a  null  user  and  therefore  cannot  track  their  activity 
on  a  site. 

Online  User:  All  users  who  come  to  an  enterprise  Intranet.  In  general,  we  can  tell  what 
type  of  browser  customers  are  using,  where  they  came  from,  and  what  areas  of  the  site 
they  visit. 

Page  Instrumentation:  The  concept  of  adding  "trigger”  code  in  the  form  of  scripts  to  a 
Web  page  that  is  able  to  relay  user  interactions  to  a  service  that  archives  the  activity  for 
analysis.  Synonym;  Click  Stream  Data  Collection. 

Page  Views:  Any  file  sent  to  a  client  that  provides  information,  such  as  http,  ASP,  Word 
documents,  etc.  Depending  on  the  business,  the  following  files  may  or  may  not  be 
counted  as  page  views;  avi,  bmp,  cdf,  class,  css,  dat,  dll,  gif,  idq,  inc,  jpg,  js,  toe,  and 
wav. 
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Page  Views  per  User:  Page  views  by  GUIDed  users  divided  by  the  number  of  unique 
users  (GUIDed).  We  use  this  number  to  indicate  how  many  pages  are  viewed  by  a  user 
on  average. 

Page  Views  by  GUIDed:  Users  The  count  of  page  views  by  GUIDed  users  only. 

Percentage  Page  Views  by  GUIDed:  Users  Page  Views  by  GUIDed  users  divided  by 
Total  page  views.  For  example,  if  there  are  10,000  pages  viewed  under  a  vroot,  with 
3,000  pages  viewed  by  GUIDed  users,  this  means  that  GUIDed  users  account  for  30%  of 
the  page  views,  and  non-GUIDed  account  for  70%  of  the  page  views. 

Persona:  If  a  business  fails  to  understand  its  users,  then  it  will  probably  create  a  poor 
product  or  service.  Personas  (or  User  Archetypes)  are  a  way  of  depicting  the  users  of 
products,  such  as  user  behavior  while  visiting  portal  sites.  Using  personas  allows 
development  teams  to  focus  more  on  design  and  apply  appropriate  effort  on  the  right  set 
of  features  to  support  these  users.  Named  and  developed  by  usability  engineers,  personas 
are  tools  that  help  make  customers  very  real  to  the  product  developers  while  comprising  a 
rich  collection  of  consumer  data,  field  research  and  other  studies.  Key  objectives  are  to 
ensure  products  are  both  useful,  so  they  do  the  things  people  want  them  to  do,  and  usable, 
so  people  can  accomplish  those  things  easily.  One  of  the  most  important  functions  of 
usability  engineers  is  to  communicate  to  the  development  team,  as  clearly  as  possible, 
who  the  users  really  are.  When  designers  and  developers  don't  share  a  distinct  image  of 
their  user,  they  carry  different  interpretations  of  'user'  around  with  them.  As  a  result,  they 
are  prone  to  developing  a  schizophrenic  interface  or  -  taken  as  a  whole  across  a  large 
enterprise  -  very  inconsistent  interfaces  that  leave  for  a  bumpy,  if  not  dissatisfying,  surf 
experience  from  portal  to  portal. 

Personalization:  A  method  of  contextualizing  information  for  a  knowledge  worker  based 
on  what  is  known  about  the  knowledge  worker.  A  site  can  use  personalization  to  alter 
navigation  and  content  presented  according  to  the  perceived  needs  of  the  knowledge 
worker.  Examples  include  content  filtering  based  on  role;  authentication  based  on  name 
or  status  as  a  manager. 

Pivot  Table:  An  interactive  data  report  that  permits  the  user  to  manipulate  the  hierarchy 
of  data  filters  as  well  as  the  organization  of  column  and  row  headings.  This  provides  a 
means  to  rearrange  data  for  different  views  during  analysis. 

Point  of  Access:  Content  can  be  reached  through  a  variety  of  points  of  access.  These 
access  points  reside  within  the  content  item  itself,  i.e.  full  text,  or  are  associated  with  the 
content  through  its  properties  and  relative  associations.  (Compare  to  the  use  of  the  term 
in  identifying  physical  points  of  access  to  an  information  system  such  as  desktop 
computers,  hand-held  computers,  and  wearable  computers.) 

Point  of  Entry:  Eogical  and  visually  identifiable  place  on  an  information  system  where  a 
knowledge  worker  can  begin  to  seek  information.  This  point  of  entry  is  then  accountable 
for  the  knowledge  worker’s  experience. 
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Policy:  Accepted  and  codified  set  of  decisions  designed  to  aoeomplish  identified 
business  goals.  Any  and  all  Web  analytie  polieies  coordinated  by  a  Business  Intelligence 
team  should  reflect  business  values. 

Portal:  A  means  into  an  information  system,  such  as  a  corporate  intranet,  established 
over  a  eoherent  body  of  information  or  community  of  interest,  also  known  as  a  site.  This 
(personalized)  Website  is  built  on  a  common  set  of  application  services  that  enables  firms 
to  conneet  employees,  customers,  partners,  and  suppliers  to  all  appropriate  eorporate 
computing  resources  from  any  Internet-eapable  device. 

Produetivity:  The  better  knowledge  workers  apply  knowledge,  the  more  productive  they 
are.  Therefore,  how  this  knowledge  is  managed  for  and  by  knowledge  workers  is  the 
most  integral,  inextrieable  aspect  of  improving  knowledge  worker  productivity. 
Productivity  benefits  from  improved  knowledge  management  include  a  eonservation  of 
resources,  maintaining  eompetitive  edge  and  ereating  new  opportunities. 

Profiled  User:  Is  a  GUIDed  user  who  has  submitted  profile  data  through  a  RegSys 
applieation.  An  un-profiled  user  is  a  GUIDed  user  who  has  not  registered. 

Query  manipulation:  The  process  of  modifying  an  original  query  to  best  match  the 
teehnology,  eontent  and  knowledge  worker  behavior  to  improve  relevaney  of  information 
presented.  Query  manipulation  ean  be  done  manually  by  the  knowledge  worker, 
interaetively  with  the  knowledge  worker  and  the  information  system  working  together  or 
automatically  by  the  seareh  teehnology.  Query  manipulation  includes  such  things  as 
word  breaking  and  expanding  a  query  term  to  include  all  of  its  synonyms. 

Raw  Logs:  Log  files  preserved  in  the  state  as  found  on  their  Web  server  of  origin.  An 
analyties  application  processes  raw  logs  using  a  server  application,  such  as  Commerce 
Server  2000  for  example,  as  it  ports  these  logs  into  the  data  warehouse. 

Reach:  Total  unique  users  for  a  specific  cluster  or  vroot  divided  by  the  total  number  of 
unique  users  for  the  entire  enterprise  domain. 

Redirect:  The  HTTP  action  of  sending  a  Web  user  to  another  physical  URL  upon  the 
request  of  another.  Redirects  are  recorded  in  the  logs  by  HTTP  status  codes  in  the  range 
of  300-399. 

Redirects  are  specific  to  whether  they  are  client-side  or  server-side.  For  example,  a  client- 
side  redirect  results  in  another  200  status  code  whereas  a  server-side  redirect  results  in  a 
302. 

Referrer:  Indicates  how  users  get  to  the  pages  they  are  on  and  which  URL  a  user  has 
come  from.  See  also  Referrer  type. 

Referrer  Type:  There  are  at  least  three  types  of  referrals  relevant  to  Web  analytics: 
external  referrer,  internal  referrer,  no  referrer 
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Referring  Stats:  This  is  the  statistic  relating  to  referrer  in  the  Web  analytics  program. 


Relative  Associations:  The  term  relative  associations  encompasses  all  types  of 
relationships  between  items  in  an  information  system.  These  relationships  are  based  on 
such  things  as  characteristics  of  the  items  (properties),  knowledge  worker  tasks  and 
interactions  between  the  items.  Once  established,  these  associations  can  be  leveraged  or 
exposed  in  an  information  system  to  enable  a  knowledge  worker  to  move  from  concept  to 
concept. 

Retention:  Percent  of  GUIDed  users  who  viewed  at  least  one  page  in  a  vroot  during  the 
previous  calendar  month  and  also  visited  at  least  one  page  in  the  same  vroot  during  the 
current  calendar  month.  All  content  items  in  an  information  system  have  value  for  a 
certain  length  of  time.  Determining  this  length  of  time  is  deciding  an  item’s  retention. 
The  length  of  time  an  item  has  value  and  is  retained  depends  on  many  factors.  An  item’s 
retention  can  be  recorded  in  metadata  and  made  available  for  information  management 
purposes. 

Roll-up:  The  summary  of  a  path  data  takes  before  it  resolves  as  a  metric  in  a  report. 

Schema:  The  representation  of  the  tables  that  depict  the  logical  organization  of  the 
database  or  data  warehouse.  It  is  the  architecture  of  an  information  storage  and  retrieval 
system,  including  the  storage  structure,  taxonomy  arrangement,  and  user-interface  design 
for  search  and  retrieval.  Particularly  important  is  the  interaction  between  the  various 
elements.  The  schema  is  the  design  and  how  the  pieces  work  together.  (Schema  includes 
metadata  schema.) 

Search:  An  application  employed  by  knowledge  workers  as  a  tool  to  find  through  direct 
surfacing  or  through  surfacing  an  obvious  navigational  path. 

Search  Stats:  Statistics  that  are  collected  specifically  on  search  engines  on  an  enterprise 
websites  that  include  the  search  strings  entered  by  users. 

(Smart)  Tags:  Constituents  of  the  schema  which  identify  and  label  metadata  elements. 
There  is  a  1:1  relationship  between  the  tags  and  the  elements.  Smart  Tags  attempt  to  do 
this  process  automatically  based  on  intelligence 

Standard:  The  processes,  protocols  and  technologies  established  or  chosen  by  Business 
Intelligence  team  as  the  models  to  follow  or  use  to  meet  policy  requirements. 

Subdirectory:  The  collection  of  zero  or  more  directories  or  files  below  a  parent  directory. 

Sub-domain:  The  term  used  to  represent  the  breakouts  of  virtual  URL  addresses  attached 
to  the  enterprise  domain.  Synonym:  Cluster. 
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Subscription:  Knowledge  worker  seleeted  delivery  of  speeifie  eontent  to  their  email 
inbox.  This  is  meant  to  occur  at  a  specified  interval,  or  as  a  result  of  an  event.  It  would 
inelude  speeifie  content  but  may  also  encompass  notification  following  the  posting  of 
related  information. 

Suceessful  Downloads:  The  number  of  sueeessful  downloads  based  on  the  HTTP  Status 
code  +  the  Win32  Code  or  Unix  Code  indicating  success. 

Surrogate:  Aeeumulation  of  metadata  representing  elasses  of  information  eommon  to  all 
members  of  a  given  content  sent.  Metadata  sehema  is  part  of  an  overall  sehema.  See: 
Sehema 

Taxonomy:  The  result  of  identifying,  ereating  and  naming  relative  assoeiations  between 
items  in  an  information  system.  In  turn,  the  relative  assoeiations  are  grouped  or  elassified 
into  taxonomic  structures  based  on  three  relationship  types:  hierarchieal,  equivalenee  and 
assoeiative.  Examples  are  eontrolled  voeabularies;  metadata  seheme;  category  labels. 
See:  Relative  Assoeiations. 

Technical  Information:  Relating  to  or  specializing  in  techniques  or  subjeets  of  applied 
seienee. 

Total  Estimated  Unique  Users:  Total  Estimated  Users  =  Total  Page  Views  /  (Page  Views 
by  GUIDed  Users  /  Total  GUIDed  Users).  This  calculation  is  made  with  the  assumption 
that  non-GUIDed  users  view  the  same  number  of  pages  as  GUIDed  users. 

Total  Page  Views:  Aggregated  number  of  page  views  by  all  users.  This  ineludes 
GUIDed  and  non-GUIDed  users. 

Total  usage  minutes:  The  total  number  of  usage  minutes  spent  at  the  website,  eategory, 
channel,  or  application  during  the  eourse  of  the  reporting  period. 

Unique  GUIDed:  User  See  Unique  Users. 

Unique  Pages:  The  count  of  individual  pages  in  a  vroot  or  eluster  whieh  have  been 
viewed  at  least  onee  in  the  period  being  measured. 

Unique  Pages  per  User:  Total  unique  pages  divided  by  the  number  of  GUIDed  users. 

Unique  Users:  The  estimated  number  of  different  individuals  (in  thousands)  that  visited 
any  eontent  of  a  website,  a  eategory,  a  ehannel,  or  an  applieation  during  the  eourse  of  the 
reporting  period.  Visitors  who  reeeive  a  GUID  by  way  of  eookie  upon  entering  portals 
within  the  enterprise  domain/website  enable  analyties  programs  to  keep  track  of  actual 
volume  of  different  visitors  to  enterprise  portals  on  a  daily,  monthly,  and  annual  basis. 
(Generally  unique  users  represent  a  non-additive  metrie  in  Web  analytie  programs,  in  that 
if  unique  users  are  analyzed  through  separate  queries  for  each  month  in  a  year,  the  sum  of 
those  queries  would  not  be  a  eorreet  representation  of  the  aetual  unique  users  for  the  year. 
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The  sum  of  the  individual  months  would  be  greater  than  the  total  unique  users  for  the 
year.) 

Unprofiled  User:  An  unprofiled  user  is  a  GUIDed  user  who  has  not  registered  online  via 
a  registration  application. 

URL:  A  Uniform  Resource  Locator  is  an  Internet  or  Intranet  address  consisting  of  the 
Internet  protocol  name,  host  name,  and  other  elements,  often  including  directory  and  file 
name. 

User  Assistance:  Help  made  available  to  the  knowledge  worker  while  using  an 
information  system.  User  assistance  provides  guidance  on  how  to  use  the  system  or 
additional  help  in  finding  information  sought.  Examples  are  help  in  context,  feedback. 
Dialog  boxes,  and  training,  real  time  assistance  from  a  human  intermediary  via  a  live 
chat. 

User  Data:  Facts  and  figures  a  knowledge  worker  maintains  private  access  to  for 
knowledge  retention  and  expansion.  This  is  also  referred  to  as  “personalization”. 

User  Experience:  The  relationship  a  knowledge  worker  has  with  the  intranet  and  the 
information  provided. 

User  Profile:  An  enterprise  profile  of  a  user  associated  with  an  enterprise  ID  system 
which  includes  specific  data  entered  by  that  user.  See  User  Type. 

User  Type:  Of  all  the  Intranet  users  who  visit  an  enterprise  portal,  a  portion  of  them  go 
to  a  registration  wizard  and  accept  an  ID  cookie  that  is  issued  to  them.  These  users  can 
now  be  counted  as  unique  users,  although  not  much  is  known  about  them  at  this  point.  If 
the  user  completes  the  basic  registration  form,  then  system  admin  has  their  e-mail  name 
and  department.  If  the  user  also  answers  more  specific  profiling  questions  on  where  they 
use  a  computer,  what  their  role  with  computers  is,  etc.,  then  they  can  be  considered 
profiled  users  and  can  be  categorize  as  a  specific  type  of  Business  User. 

Vroot:  Abbreviation  for  Virtual  Root,  which  is  the  logical  root  for  a  website  on  the 
Enterprise  domain.  Typically  the  default  page  for  a  website  will  be  stored  directly  under 
the  vroot. 

Web  Service:  A  Web  Service  is  a  unit  of  application  logic  providing  data  and  services  to 
other  applications.  Applications  access  Web  Services  via  ubiquitous  Web  protocols  and 
data  formats  such  as  HTTP,  XML,  and  SOAP,  without  worry  about  how  each  Web 
service  is  implemented.  Web  services  combine  the  best  aspects  of  component-based 
development  and  the  Web. 
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